Acceleration of Expectation-Maximization algorithm for length-biased right-censored data

Vardi’s Expectation-Maximization (EM) algorithm is frequently used for computing the nonparametric maximum likelihood estimator of length-biased right-censored data, which does not admit a closed-form representation. The EM algorithm may converge slowly, particularly for heavily censored data. We studied two algorithms for accelerating the convergence of the EM algorithm, based on iterative convex minorant and Aitken’s delta squared process. Numerical simulations demonstrate that the acceleration algorithms converge more rapidly than the EM algorithm in terms of number of iterations and actual timing. The acceleration method based on a modification of Aitken’s delta squared performed the best under a variety of settings.     

Standard errors for EM estimation

The EM algorithm is a popular method for computing maximum likelihood estimates. One of its drawbacks is that it does not produce standard errors as a by-product. We consider obtaining standard errors by numerical differentiation. Two approaches are considered. The first differentiates the Fisher score vector to yield the Hessian of the log-likelihood. The second differentiates the EM operator and uses an identity that relates its derivative to the Hessian of the log-likelihood. The well-known SEM algorithm uses the second approach. We consider three additional algorithms: one that uses the first approach and two that use the second. We evaluate the complexity and precision of these three and the SEM in algorithm seven examples. The first is a single-parameter example used to give insight. The others are three examples in each of two areas of EM application: Poisson mixture models and the estimation of covariance from incomplete data. The examples show that there are algorithms that are much simpler and more accurate than the SEM algorithm. Hopefully their simplicity will increase the availability of standard error estimates in EM applications. It is shown that, as previously conjectured, a symmetry diagnostic can accurately estimate errors arising from numerical differentiation. Some issues related to the speed of the EM algorithm and algorithms that differentiate the EM operator are identified.     

Quantile Regression via the EM Algorithm

The three-parameter asymmetric Laplace distribution (ALD) has received increasing attention in the field of quantile regression due to an important feature between its location and asymmetric parameters. On the basis of the representation of the ALD as a normal-variance–mean mixture with an exponential mixing distribution, this article develops EM and generalized EM algorithms, respectively, for computing regression quantiles of linear and nonlinear regression models. It is interesting to show that the proposed EM algorithm and the MM (Majorization–Minimization) algorithm for quantile regressions are really the same in terms of computation, since the updating formula of them are the same. This provides a good example that connects the EM and MM algorithms. Simulation studies show that the EM algorithm can successfully recover the true parameters in quantile regressions.     

EM algorithm for mixture of skew-normal distributions fitted to grouped data

Grouped data are frequently used in several fields of study. In this work, we use the expectation-maximization (EM) algorithm for fitting the skew-normal (SN) mixture model to the grouped data. Implementing the EM algorithm requires computing the one-dimensional integrals for each group or class. Our simulation study and real data analyses reveal that the EM algorithm not only always converges but also can be implemented in just a few seconds even when the number of components is large, contrary to the Bayesian paradigm that is computationally expensive. The accuracy of the EM algorithm and superiority of the SN mixture model over the traditional normal mixture model in modelling grouped data are demonstrated through the simulation and three real data illustrations. For implementing the EM algorithm, we use the package called ForestFit developed for R environment available at https://cran.r-project.org/web/packages/ForestFit/index.html.     

Self-updating clustering algorithm for estimating the parameters in mixtures of von Mises distributions

The EM algorithm is the standard method for estimating the parameters in finite mixture models. Yang and Pan [25] proposed a generalized classification maximum likelihood procedure, called the fuzzy c-directions (FCD) clustering algorithm, for estimating the parameters in mixtures of von Mises distributions. Two main drawbacks of the EM algorithm are its slow convergence and the dependence of the solution on the initial value used. The choice of initial values is of great importance in the algorithm-based literature as it can heavily influence the speed of convergence of the algorithm and its ability to locate the global maximum. On the other hand, the algorithmic frameworks of EM and FCD are closely related. Therefore, the drawbacks of FCD are the same as those of the EM algorithm. To resolve these problems, this paper proposes another clustering algorithm, which can self-organize local optimal cluster numbers without using cluster validity functions. These numerical results clearly indicate that the proposed algorithm is superior in performance of EM and FCD algorithms. Finally, we apply the proposed algorithm to two real data sets.     

Nonparametric Random Variate Generation Using a Piecewise-Linear Cumulative Distribution Function

The standard approach to solving the interpolation problem for a trace-driven simulation involving a continuous random variable is to construct a piecewise-linear cdf that fills in the gaps between the data values. Some probabilistic properties of this estimator are derived, and three extensions to the standard approach (matching moments, weighted values, and right-censored data) are presented, along with associated random variate generation algorithms. The algorithm is a nonparametric blackbox variate generator requiring only observed data from the user.     

Identity for the NPMLE in Censored Data Models

We derive an identity for nonparametric maximum likelihood estimators (NPMLE) and regularized MLEs in censored data models which expresses the standardized maximum likelihood estimator in terms of the standardized empirical process. This identity provides an effective starting point in proving both consistency and efficiency of NPMLE and regularized MLE. The identity and corresponding method for proving efficiency is illustrated for the NPMLE in the univariate right-censored data model, the regularized MLE in the current status data model and for an implicit NPMLE based on a mixture of right-censored and current status data. Furthermore, a general algorithm for estimation of the limiting variance of the NPMLE is provided. This revised version was published online in July 2006 with corrections to the Cover Date.     

Fitting mixtures of linear regressions

In most applications, the parameters of a mixture of linear regression models are estimated by maximum likelihood using the expectation maximization (EM) algorithm. In this article, we propose the comparison of three algorithms to compute maximum likelihood estimates of the parameters of these models: the EM algorithm, the classification EM algorithm and the stochastic EM algorithm. The comparison of the three procedures was done through a simulation study of the performance (computational effort, statistical properties of estimators and goodness of fit) of these approaches on simulated data sets.

Simulation results show that the choice of the approach depends essentially on the configuration of the true regression lines and the initialization of the algorithms.     

A Poisson-multinomial mixture approach to grouped and right-censored counts

Although count data are often collected in social, psychological, and epidemiological surveys in grouped and right-censored categories, there is a lack of statistical methods simultaneously taking both grouping and right-censoring into account. In this research, we propose a new generalized Poisson-multinomial mixture approach to model grouped and right-censored (GRC) count data. Based on a mixed Poisson-multinomial process for conceptualizing grouped and right-censored count data, we prove that the new maximum-likelihood estimator (MLE-GRC) is consistent and asymptotically normally distributed for both Poisson and zero-inflated Poisson models. The use of the MLE-GRC, implemented in an R function, is illustrated by both statistical simulation and empirical examples. This research provides a tool for epidemiologists to estimate incidence from grouped and right-censored count data and lays a foundation for regression analyses of such data structure.     

Simulation of right and left truncated gamma distributions by mixtures

We study the properties of truncated gamma distributions and we derive simulation algorithms which dominate the standard algorithms for these distributions. For the right truncated gamma distribution, an optimal accept–reject algorithm is based on the fact that its density can be expressed as an infinite mixture of beta distribution. For integer values of the parameters, the density of the left truncated distributions can be rewritten as a mixture which can be easily generated. We give an optimal accept–reject algorithm for the other values of the parameter. We compare the efficiency of our algorithm with the previous method and show the improvement in terms of minimum acceptance probability. The algorithm proposed here has an acceptance probability which is superior to e/4.     

Review and implementation of cure models based on first hitting times for Wiener processes

The development of models and methods for cure rate estimation has recently burgeoned into an important subfield of survival analysis. Much of the literature focuses on the standard mixture model. Recently, process-based models have been suggested. We focus on several models based on first passage times for Wiener processes. Whitmore and others have studied these models in a variety of contexts. Lee and Whitmore (Stat Sci 21(4):501–513, 2006) give a comprehensive review of a variety of first hitting time models and briefly discuss their potential as cure rate models. In this paper, we study the Wiener process with negative drift as a possible cure rate model but the resulting defective inverse Gaussian model is found to provide a poor fit in some cases. Several possible modifications are then suggested, which improve the defective inverse Gaussian. These modifications include: the inverse Gaussian cure rate mixture model; a mixture of two inverse Gaussian models; incorporation of heterogeneity in the drift parameter; and the addition of a second absorbing barrier to the Wiener process, representing an immunity threshold. This class of process-based models is a useful alternative to the standard model and provides an improved fit compared to the standard model when applied to many of the datasets that we have studied. Implementation of this class of models is facilitated using expectation-maximization (EM) algorithms and variants thereof, including the gradient EM algorithm. Parameter estimates for each of these EM algorithms are given and the proposed models are applied to both real and simulated data, where they perform well.     

Mini-batch learning of exponential family finite mixture models

Mini-batch algorithms have become increasingly popular due to the requirement for solving optimization problems, based on large-scale data sets. Using an existing online expectation–maximization (EM) algorithm framework, we demonstrate how mini-batch (MB) algorithms may be constructed, and propose a scheme for the stochastic stabilization of the constructed mini-batch algorithms. Theoretical results regarding the convergence of the mini-batch EM algorithms are presented. We then demonstrate how the mini-batch framework may be applied to conduct maximum likelihood (ML) estimation of mixtures of exponential family distributions, with emphasis on ML estimation for mixtures of normal distributions. Via a simulation study, we demonstrate that the mini-batch algorithm for mixtures of normal distributions can outperform the standard EM algorithm. Further evidence of the performance of the mini-batch framework is provided via an application to the famous MNIST data set.     

Asymptotic properties of a stochastic EM algorithm for mixtures with censored data

Weak consistency and asymptotic normality is shown for a stochastic EM algorithm for censored data from a mixture of distributions under lognormal assumptions. The asymptotic properties hold for all parameters of the distributions, including the mixing parameter. In order to make parameter estimation meaningful it is necessary to know that the censored mixture distribution is identifiable. General conditions under which this is the case are given. The stochastic EM algorithm addressed in this paper is used for estimation of wood fibre length distributions based on optically measured data from cylindric wood samples (increment cores).     

Maximum likelihood estimation under a finite mixture of generalized exponential distributions based on censored data

In this paper, the identifiability of a finite mixture of generalized exponential distributions (GE(τ, α)) is proved and the maximum likelihood estimates (MLE’s) of the parameters are obtained using EM algorithm based on a general form of right-censored failure times. The results are specialized to type-I and type-II censored samples. A real data set is introduced and analyzed using a mixture of two GE(τ, α) distributions and also using a mixture of two Weibull(α, β) distributions. A comparison is carried out between the mentioned mixtures based on the corresponding Kolmogorov–Smirnov (K–S) test statistic to emphasize that the GE(τ, α) mixture model fits the data better than the other mixture model.     

A new REML (parameter expanded) EM algorithm for linear mixed models

Linear mixed models are regularly applied to animal and plant breeding data to evaluate genetic potential. Residual maximum likelihood (REML) is the preferred method for estimating variance parameters associated with this type of model. Typically an iterative algorithm is required for the estimation of variance parameters. Two algorithms which can be used for this purpose are the expectation‐maximisation (EM) algorithm and the parameter expanded EM (PX‐EM) algorithm. Both, particularly the EM algorithm, can be slow to converge when compared to a Newton‐Raphson type scheme such as the average information (AI) algorithm. The EM and PX‐EM algorithms require specification of the complete data, including the incomplete and missing data. We consider a new incomplete data specification based on a conditional derivation of REML. We illustrate the use of the resulting new algorithm through two examples: a sire model for lamb weight data and a balanced incomplete block soybean variety trial. In the cases where the AI algorithm failed, a REML PX‐EM based on the new incomplete data specification converged in 28% to 30% fewer iterations than the alternative REML PX‐EM specification. For the soybean example a REML EM algorithm using the new specification converged in fewer iterations than the current standard specification of a REML PX‐EM algorithm. The new specification integrates linear mixed models, Henderson's mixed model equations, REML and the REML EM algorithm into a cohesive framework.     

An em algorithm for a semiparametric finite mixture model

We propose a semiparametric version of the EM algorithm under the semiparametric mixture model introduced by Anderson (1979, Biometrika , 66 , 17-26). It is shown that the sequence of proposed EM iterates, irrespective of the starting value, converges to the maximum semiparametric likelihood estimator of the vector of parameters in the semiparametric mixture model. The proposed EM algorithm preserves the appealing monotone convergence property of the standard EM algorithm and can be implemented by employing the standard logistic regression program. We present one example to demonstrate the performance of the proposed EM algorithm.     

Trajectory Modeling of Longitudinal Binary Data: Application of the EM Algorithm for Mixture Models

A developmental trajectory describes the course of behavior over time. Identifying multiple trajectories within an overall developmental process permits a focus on subgroups of particular interest. We introduce a framework for identifying trajectories by using the Expectation-Maximization (EM) algorithm to fit semiparametric mixtures of logistic distributions to longitudinal binary data. For performance comparison, we consider full maximization algorithms (PROC TRAJ in SAS), standard EM, and two other EM-based algorithms for speeding up convergence. Simulation shows that EM methods produce more accurate parameter estimates. The EM methodology is illustrated with a longitudinal dataset involving adolescents smoking behaviors.     

An online classification EM algorithm based on the mixture model

Mixture model-based clustering is widely used in many applications. In certain real-time applications the rapid increase of data size with time makes classical clustering algorithms too slow. An online clustering algorithm based on mixture models is presented in the context of a real-time flaw-diagnosis application for pressurized containers which uses data from acoustic emission signals. The proposed algorithm is a stochastic gradient algorithm derived from the classification version of the EM algorithm (CEM). It provides a model-based generalization of the well-known online k-means algorithm, able to handle non-spherical clusters. Using synthetic and real data sets, the proposed algorithm is compared with the batch CEM algorithm and the online EM algorithm. The three approaches generate comparable solutions in terms of the resulting partition when clusters are relatively well separated, but online algorithms become faster as the size of the available observations increases.     

Fitting finite mixture models using iterative Monte Carlo classification

Parameters of a finite mixture model are often estimated by the expectation–maximization (EM) algorithm where the observed data log-likelihood function is maximized. This paper proposes an alternative approach for fitting finite mixture models. Our method, called the iterative Monte Carlo classification (IMCC), is also an iterative fitting procedure. Within each iteration, it first estimates the membership probabilities for each data point, namely the conditional probability of a data point belonging to a particular mixing component given that the data point value is obtained, it then classifies each data point into a component distribution using the estimated conditional probabilities and the Monte Carlo method. It finally updates the parameters of each component distribution based on the classified data. Simulation studies were conducted to compare IMCC with some other algorithms for fitting mixture normal, and mixture t, densities.     

Multivariate normal mean-variance mixture distribution based on Lindley distribution

This article introduces a new asymmetric distribution constructed by assuming the multivariate normal mean-variance mixture model. Called normal mean-variance mixture of the Lindley distribution, we derive some mathematical properties of the new distribution. Also, a feasible maximum likelihood estimation procedure using the EM algorithm and the asymptotic standard errors of parameter estimates are developed. The performance of the proposed distribution is illustrated by means of real datasets and simulation analysis.