Variable Selection for Partially Linear Models with Randomly Censored Data

This article proposes a variable selection procedure for partially linear models with right-censored data via penalized least squares. We apply the SCAD penalty to select significant variables and estimate unknown parameters simultaneously. The sampling properties for the proposed procedure are investigated. The rate of convergence and the asymptotic normality of the proposed estimators are established. Furthermore, the SCAD-penalized estimators of the nonzero coefficients are shown to have the asymptotic oracle property. In addition, an iterative algorithm is proposed to find the solution of the penalized least squares. Simulation studies are conducted to examine the finite sample performance of the proposed method.     

Strong Consistency of the Maximum Likelihood Estimator for Finite Mixtures of Location–Scale Distributions When Penalty is Imposed on the Ratios of the Scale Parameters

Abstract.  In finite mixtures of location–scale distributions, if there is no constraint or penalty on the parameters, then the maximum likelihood estimator does not exist because the likelihood is unbounded. To avoid this problem, we consider a penalized likelihood, where the penalty is a function of the minimum of the ratios of the scale parameters and the sample size. It is shown that the penalized maximum likelihood estimator is strongly consistent. We also analyse the consistency of a penalized maximum likelihood estimator where the penalty is imposed on the scale parameters themselves.     

Large sample interval mapping method for genetic trait loci in finite regression mixture models

This article investigates the large sample interval mapping method for genetic trait loci (GTL) in a finite non-linear regression mixture model. The general model includes most commonly used kernel functions, such as exponential family mixture, logistic regression mixture and generalized linear mixture models, as special cases. The populations derived from either the backcross or intercross design are considered. In particular, unlike all existing results in the literature in the finite mixture models, the large sample results presented in this paper do not require the boundness condition on the parametric space. Therefore, the large sample theory presented in this article possesses general applicability to the interval mapping method of GTL in genetic research. The limiting null distribution of the likelihood ratio test statistics can be utilized easily to determine the threshold values or p-values required in the interval mapping. The limiting distribution is proved to be free of the parameter values of null model and free of the choice of a kernel function. Extension to the multiple marker interval GTL detection is also discussed. Simulation study results show favorable performance of the asymptotic procedure when sample sizes are moderate.     

Generalized additive models for longitudinal data

We introduce a class of models for longitudinal data by extending the generalized estimating equations approach of Liang and Zeger (1986) to incorporate the flexibility of nonparametric smoothing. The algorithm provides a unified estimation procedure for marginal distributions from the exponential family. We propose pointwise standard-error bands and approximate likelihood-ratio and score tests for inference. The algorithm is formally derived by using the penalized quasilikelihood framework. Convergence of the estimating equations and consistency of the resulting solutions are discussed. We illustrate the algorithm with data on the population dynamics of Colorado potato beetles on potato plants.     

A new model selection procedure for finite mixture regression models

Abstract

In this article, we propose a new penalized-likelihood method to conduct model selection for finite mixture of regression models. The penalties are imposed on mixing proportions and regression coefficients, and hence order selection of the mixture and the variable selection in each component can be simultaneously conducted. The consistency of order selection and the consistency of variable selection are investigated. A modified EM algorithm is proposed to maximize the penalized log-likelihood function. Numerical simulations are conducted to demonstrate the finite sample performance of the estimation procedure. The proposed methodology is further illustrated via real data analysis.     

Testing homogeneity in a mixture of von mises distributions with a structural parameter

The modified likelihood ratio statistic can be used to test the homogeneity in a variety of mixture models. Here, the authors propose the use of the modified and the iterative modified likelihood ratio for testing homogeneity against a two‐component von Mises mixture with a structural parameter. They derive the limiting distributions of the test statistics and propose methods to improve the accuracy of the asymptotic approximation in finite samples. Their simulations show that the tests maintain their nominal level and that they have adequate power. Data on movements of turtles are used as an illustration     

Variable Selection for Semiparametric Partially Linear Covariate-Adjusted Regression Models

In this article, the partially linear covariate-adjusted regression models are considered, and the penalized least-squares procedure is proposed to simultaneously select variables and estimate the parametric components. The rate of convergence and the asymptotic normality of the resulting estimators are established under some regularization conditions. With the proper choices of the penalty functions and tuning parameters, it is shown that the proposed procedure can be as efficient as the oracle estimators. Some Monte Carlo simulation studies and a real data application are carried out to assess the finite sample performances for the proposed method.     

Model selection for the localized mixture of experts models

In this paper, we propose a penalized likelihood method to simultaneous select covariate, and mixing component and obtain parameter estimation in the localized mixture of experts models. We develop an expectation maximization algorithm to solve the proposed penalized likelihood procedure, and introduce a data-driven procedure to select the tuning parameters. Extensive numerical studies are carried out to compare the finite sample performances of our proposed method and other existing methods. Finally, we apply the proposed methodology to analyze the Boston housing price data set and the baseball salaries data set.     

Estimating curves and derivatives with parametric penalized spline smoothing

Accurate estimation of an underlying function and its derivatives is one of the central problems in statistics. Parametric forms are often proposed based on the expert opinion or prior knowledge of the underlying function. However, these strict parametric assumptions may result in biased estimates when they are not completely accurate. Meanwhile, nonparametric smoothing methods, which do not impose any parametric form, are quite flexible. We propose a parametric penalized spline smoothing method, which has the same flexibility as the nonparametric smoothing methods. It also uses the prior knowledge of the underlying function by defining an additional penalty term using the distance of the fitted function to the assumed parametric function. Our simulation studies show that the parametric penalized spline smoothing method can obtain more accurate estimates of the function and its derivatives than the penalized spline smoothing method. The parametric penalized spline smoothing method is also demonstrated by estimating the human height function and its derivatives from the real data.     

Removing the singularity of a penalty via thresholding function matching

By introducing the idea of thresholding function matching, it is illustrated that both bridge penalty and log penalty can be transformed so as to circumvent certain difficulties in numerical computation and the definition of local minimality. The fact that both bridge penalty and log penalty have derivatives going to infinity at zero. This hinders their applications in statistics although it is reported in the literature that they allow recovery of sparse structure in the data under some conditions. It is illustrated in the simulation studies that in the variable selection problems, penalized likelihood estimation based on the transformed penalty obtained by the proposed thresholding function matching method outperform those based on many other state-of-art penalties, particularly when the covariates are strongly correlated. The one-to-one correspondence between the transformed penalties and their thresholding functions are also established.     

Likelihood ratio tests in linear mixed models with one variance component

Summary.  We consider the problem of testing null hypotheses that include restrictions on the variance component in a linear mixed model with one variance component and we derive the finite sample and asymptotic distribution of the likelihood ratio test and the restricted likelihood ratio test. The spectral representations of the likelihood ratio test and the restricted likelihood ratio test statistics are used as the basis of efficient simulation algorithms of their null distributions. The large sample χ ² mixture approximations using the usual asymptotic theory for a null hypothesis on the boundary of the parameter space have been shown to be poor in simulation studies. Our asymptotic calculations explain these empirical results. The theory of Self and Liang applies only to linear mixed models for which the data vector can be partitioned into a large number of independent and identically distributed subvectors. One-way analysis of variance and penalized splines models illustrate the results.     

Penalized minimum-distance estimates in finite mixture models

When finite mixture models are used to fit data, it is sometimes important to estimate the number of mixture components. A nonparametric maximum-likelihood approach may result in too many support points and, in general, does not yield a consistent estimator. A penalized likelihood approach tends to produce a fit with fewer components, but it is not known whether that approach produces a consistent estimate of the number of mixture components. We suggest the use of a penalized minimum-distance method. It is shown that the estimator obtained is consistent for both the mixing distribution and the number of mixture components.     

Divergences and duality for estimation and test under moment condition models

We introduce estimation and test procedures through divergence minimization for models satisfying linear constraints with unknown parameter. These procedures extend the empirical likelihood (EL) method and share common features with generalized empirical likelihood approach. We treat the problems of existence and characterization of the divergence projections of probability distributions on sets of signed finite measures. We give a precise characterization of duality, for the proposed class of estimates and test statistics, which is used to derive their limiting distributions (including the EL estimate and the EL ratio statistic) both under the null hypotheses and under alternatives or misspecification. An approximation to the power function is deduced as well as the sample size which ensures a desired power for a given alternative.     

Simultaneous sparse model selection and coefficient estimation for heavy-tailed autoregressive processes

We propose a sparse coefficient estimation and automated model selection procedure for autoregressive processes with heavy-tailed innovations based on penalized conditional maximum likelihood. Under mild moment conditions on the innovation processes, the penalized conditional maximum likelihood estimator satisfies a strong consistency, O_P(N^?1/2) consistency, and the oracle properties, where N is the sample size. We have the freedom in choosing penalty functions based on the weak conditions on them. Two penalty functions, least absolute shrinkage and selection operator and smoothly clipped average deviation, are compared. The proposed method provides a distribution-based penalized inference to AR models, which is especially useful when the other estimation methods fail or under perform for AR processes with heavy-tailed innovations [Feigin, Resnick. Pitfalls of fitting autoregressive models for heavy-tailed time series. Extremes. 1999;1:391–422]. A simulation study confirms our theoretical results. At the end, we apply our method to a historical price data of the US Industrial Production Index for consumer goods, and obtain very promising results.     

A Note on the Modified Likelihood Ratio Test for Homogeneity in Beta Mixture Models

This article presents a note on the modified likelihood ratio test for homogeneity in beta mixture models. Under consistency of the penalized maximum likelihood estimators, the limiting distribution of the test statistic converges to the chi-bar-squared distributions. The statistic degenerates to zero with a weight due to the negative definiteness of a complicated random matrix. The probability that this matrix is negative definite is related to the parameter values under the homogeneity hypothesis. The dependency pattern enables the introduction of an upper bound on the asymptotic null distribution. Simulation study is investigated to verify the accuracy of the results.     

Majorization minimization by coordinate descent for concave penalized generalized linear models

Recent studies have demonstrated theoretical attractiveness of a class of concave penalties in variable selection, including the smoothly clipped absolute deviation and minimax concave penalties. The computation of the concave penalized solutions in high-dimensional models, however, is a difficult task. We propose a majorization minimization by coordinate descent (MMCD) algorithm for computing the concave penalized solutions in generalized linear models. In contrast to the existing algorithms that use local quadratic or local linear approximation to the penalty function, the MMCD seeks to majorize the negative log-likelihood by a quadratic loss, but does not use any approximation to the penalty. This strategy makes it possible to avoid the computation of a scaling factor in each update of the solutions, which improves the efficiency of coordinate descent. Under certain regularity conditions, we establish theoretical convergence property of the MMCD. We implement this algorithm for a penalized logistic regression model using the SCAD and MCP penalties. Simulation studies and a data example demonstrate that the MMCD works sufficiently fast for the penalized logistic regression in high-dimensional settings where the number of covariates is much larger than the sample size.     

Estimation and variable selection for partially functional linear models

In this paper, a new estimation procedure based on composite quantile regression and functional principal component analysis (PCA) method is proposed for the partially functional linear regression models (PFLRMs). The proposed estimation method can simultaneously estimate both the parametric regression coefficients and functional coefficient components without specification of the error distributions. The proposed estimation method is shown to be more efficient empirically for non-normal random error, especially for Cauchy error, and almost as efficient for normal random errors. Furthermore, based on the proposed estimation procedure, we use the penalized composite quantile regression method to study variable selection for parametric part in the PFLRMs. Under certain regularity conditions, consistency, asymptotic normality, and Oracle property of the resulting estimators are derived. Simulation studies and a real data analysis are conducted to assess the finite sample performance of the proposed methods.     

Improving logistic regression on the imbalanced data by a novel penalized log-likelihood function

Logistic regression is estimated by maximizing the log-likelihood objective function formulated under the assumption of maximizing the overall accuracy. That does not apply to the imbalanced data. The resulting models tend to be biased towards the majority class (i.e. non-event), which can bring great loss in practice. One strategy for mitigating such bias is to penalize the misclassification costs of observations differently in the log-likelihood function. Existing solutions require either hard hyperparameter estimating or high computational complexity. We propose a novel penalized log-likelihood function by including penalty weights as decision variables for observations in the minority class (i.e. event) and learning them from data along with model coefficients. In the experiments, the proposed logistic regression model is compared with the existing ones on the statistics of area under receiver operating characteristics (ROC) curve from 10 public datasets and 16 simulated datasets, as well as the training time. A detailed analysis is conducted on an imbalanced credit dataset to examine the estimated probability distributions, additional performance measurements (i.e. type I error and type II error) and model coefficients. The results demonstrate that both the discrimination ability and computation efficiency of logistic regression models are improved using the proposed log-likelihood function as the learning objective.     

Finite Mixture of Generalized Semiparametric Models: Variable Selection via Penalized Estimation

Selection of the important variables is one of the most important model selection problems in statistical applications. In this article, we address variable selection in finite mixture of generalized semiparametric models. To overcome computational burden, we introduce a class of variable selection procedures for finite mixture of generalized semiparametric models using penalized approach for variable selection. Estimation of nonparametric component will be done via multivariate kernel regression. It is shown that the new method is consistent for variable selection and the performance of proposed method will be assessed via simulation.     

Estimation of a nonlinear discriminant function from a mixture of two GEV distributions

In this paper, the identifiability of finite mixture of generalized extreme value (GEV) distributions is proved. Next, a procedure for finding maximum likelihood estimates (MLEs) of the parameters of a finite mixture of two generalized extreme value (MGEV) distributions is presented by using classified and unclassified observations. Then, a nonlinear discriminant function for a mixture of two GEV distributions is derived and the performance of the corresponding estimated discriminant function is investigated through a series of simulation experiments. Finally, the methodology is applied to real data.