Degeneracy in the Maximum Likelihood Estimation of Univariate Gaussian Mixtures for Grouped Data and Behaviour of the EM Algorithm

Abstract.  In the context of the univariate Gaussian mixture with grouped data, it is shown that the global maximum of the likelihood may correspond to a situation where a Dirac lies in any non-empty interval. Existence of a domain of attraction near such a maximizer is discussed and we establish that the expectation-maximization (EM) iterates move extremely slowly inside this domain. These theoretical results are illustrated both by some Monte-Carlo experiments and by a real data set. To help practitioners identify and discard these potentially dangerous degenerate maximizers, a specific stopping rule for EM is proposed.     

Simple and Globally Convergent Methods for Accelerating the Convergence of Any EM Algorithm

Abstract.  The expectation-maximization (EM) algorithm is a popular approach for obtaining maximum likelihood estimates in incomplete data problems because of its simplicity and stability (e.g. monotonic increase of likelihood). However, in many applications the stability of EM is attained at the expense of slow, linear convergence. We have developed a new class of iterative schemes, called squared iterative methods (SQUAREM), to accelerate EM, without compromising on simplicity and stability. SQUAREM generally achieves superlinear convergence in problems with a large fraction of missing information. Globally convergent schemes are easily obtained by viewing SQUAREM as a continuation of EM. SQUAREM is especially attractive in high-dimensional problems, and in problems where model-specific analytic insights are not available. SQUAREM can be readily implemented as an 'off-the-shelf' accelerator of any EM-type algorithm, as it only requires the EM parameter updating. We present four examples to demonstrate the effectiveness of SQUAREM. A general-purpose implementation (written in R) is available.     

On the Optimality of Multivariate S‐Estimators

Abstract. In this article, we maximize the efficiency of a multivariate S‐estimator under a constraint on the breakdown point. In the linear regression model, it is known that the highest possible efficiency of a maximum breakdown S‐estimator is bounded above by 33 per cent for Gaussian errors. We prove the surprising result that in dimensions larger than one, the efficiency of a maximum breakdown S‐estimator of location and scatter can get arbitrarily close to 100 per cent, by an appropriate selection of the loss function.