Standard errors for EM estimation

The EM algorithm is a popular method for computing maximum likelihood estimates. One of its drawbacks is that it does not produce standard errors as a by-product. We consider obtaining standard errors by numerical differentiation. Two approaches are considered. The first differentiates the Fisher score vector to yield the Hessian of the log-likelihood. The second differentiates the EM operator and uses an identity that relates its derivative to the Hessian of the log-likelihood. The well-known SEM algorithm uses the second approach. We consider three additional algorithms: one that uses the first approach and two that use the second. We evaluate the complexity and precision of these three and the SEM in algorithm seven examples. The first is a single-parameter example used to give insight. The others are three examples in each of two areas of EM application: Poisson mixture models and the estimation of covariance from incomplete data. The examples show that there are algorithms that are much simpler and more accurate than the SEM algorithm. Hopefully their simplicity will increase the availability of standard error estimates in EM applications. It is shown that, as previously conjectured, a symmetry diagnostic can accurately estimate errors arising from numerical differentiation. Some issues related to the speed of the EM algorithm and algorithms that differentiate the EM operator are identified.     

COVARIANCE MODELS FOR LATTICE DATA

Given observations on an m × n lattice, approximate maximum likelihood estimates are derived for a family of models including direct covariance, spatial moving average, conditional autoregressive and simultaneous autoregressive models. The approach involves expressing the (approximate) covariance matrix of the observed variables in terms of a linear combination of neighbour relationship matrices, raised to a power. The structure is such that the eigenvectors of the covariance matrix are independent of the parameters of interest. This result leads to a simple Fisher scoring type algorithm for estimating the parameters. The ideas are illustrated by fitting models to some remotely sensed data.     

A monte carlo comparison of the smoothing,scoring and em algorithms for dispersion matrix estimation with incomplete growth curve data

Incomplete growth curve data often result from missing or mistimed observations in a repeated measures design. Virtually all methods of analysis rely on the dispersion matrix estimates. A Monte Carlo simulation was used to compare three methods of estimation of dispersion matrices for incomplete growth curve data. The three methods were: 1) maximum likelihood estimation with a smoothing algorithm, which finds the closest positive semidefinite estimate of the pairwise estimated dispersion matrix; 2) a mixed effects model using the EM (estimation maximization) algorithm; and 3) a mixed effects model with the scoring algorithm. The simulation included 5 dispersion structures, 20 or 40 subjects with 4 or 8 observations per subject and 10 or 30% missing data. In all the simulations, the smoothing algorithm was the poorest estimator of the dispersion matrix. In most cases, there were no significant differences between the scoring and EM algorithms. The EM algorithm tended to be better than the scoring algorithm when the variances of the random effects were close to zero, especially for the simulations with 4 observations per subject and two random effects.     

Identifiability and Comparison of Estimation Methods on Weibull Mixture Models

In this article, the finite mixture model of Weibull distributions is studied, the identifiability of the model with m components is proven, and the parameter estimators for the case of two components resulted by several algorithms are compared. The parameter estimators are obtained with maximum likelihood performing calculations with different algorithms: expectation-maximization (EM), Fisher scoring, backfitting, optimization of k-nearest neighbor approach, and random walk algorithm using Monte Carlo simulation. The Akaike information criterion and the log-likelihood value are used to compare models. In general, the proposed random walk algorithm shows better performance in mean square error and bias. Finally, the results are applied to electronic component lifetime data.     

Estimating the propagation rate of a viral infection of potato plants via mixtures of regressions

A problem arising from the study of the spread of a viral infection among potato plants by aphids appears to involve a mixture of two linear regressions on a single predictor variable. The plant scientists studying the problem were particularly interested in obtaining a 95% confidence upper bound for the infection rate. We discuss briefly the procedure for fitting mixtures of regression models by means of maximum likelihood, effected via the EM algorithm. We give general expressions for the implementation of the M-step and then address the issue of conducting statistical inference in this context. A technique due to T. A. Louis may be used to estimate the covariance matrix of the parameter estimates by calculating the observed Fisher information matrix. We develop general expressions for the entries of this information matrix. Having the complete covariance matrix permits the calculation of confidence and prediction bands for the fitted model. We also investigate the testing of hypotheses concerning the number of components in the mixture via parametric and 'semiparametric' bootstrapping. Finally, we develop a method of producing diagnostic plots of the residuals from a mixture of linear regressions.     

Gaussian mixture analysis of covariance

ABSTRACT

In many real-world applications, the traditional theory of analysis of covariance (ANCOVA) leads to inadequate and unreliable results because of violation of the response variable observations from the essential Gaussian assumption that may be due to the heterogeneity of population, the presence of outlier or both of them. In this paper, we develop a Gaussian mixture ANCOVA model for modelling heterogeneous populations with a finite number of subpopulation. We provide the maximum likelihood estimates of the model parameters via an EM algorithm. We also drive the adjusted effects estimators for treatments and covariates. The Fisher information matrix of the model and asymptotic confidence intervals for the parameter are also discussed. We performed a simulation study to assess the performance of the proposed model. A real-world example is also worked out to explained the methodology.     

基于EM算法的改进MLM模型及参数估计

传统的分层模型假设组与组之间独立,没有考虑组之间的相关性。而以地理单元分组的数据往往具有空间依赖性,个体不仅受本地区的影响,也可能受相邻地区的影响。此时,传统分层模型层-2残差分布的假设不再成立。为了处理空间分层数据,将空间统计和空间计量经济模型的思想引入到分层模型中,既纳入分层的思想,又顾及空间相关性,提出了空间分层线性模型,并给出了其固定效应、方差协方差成分和空间回归参数的最大似然估计,在运用EM算法时,结合运用了Fisher得分算法。     

On the estimation of the extreme value and normal distribution parameters based on progressive type-II hybrid-censored data

A progressive hybrid censoring scheme is a mixture of type-I and type-II progressive censoring schemes. In this paper, we mainly consider the analysis of progressive type-II hybrid-censored data when the lifetime distribution of the individual item is the normal and extreme value distributions. Since the maximum likelihood estimators (MLEs) of these parameters cannot be obtained in the closed form, we propose to use the expectation and maximization (EM) algorithm to compute the MLEs. Also, the Newton–Raphson method is used to estimate the model parameters. The asymptotic variance–covariance matrix of the MLEs under EM framework is obtained by Fisher information matrix using the missing information and asymptotic confidence intervals for the parameters are then constructed. This study will end up with comparing the two methods of estimation and the asymptotic confidence intervals of coverage probabilities corresponding to the missing information principle and the observed information matrix through a simulation study, illustrated examples and real data analysis.     

Likelihood inference for lognormal data with left truncation and right censoring with an illustration

The lognormal distribution is quite commonly used as a lifetime distribution. Data arising from life-testing and reliability studies are often left truncated and right censored. Here, the EM algorithm is used to estimate the parameters of the lognormal model based on left truncated and right censored data. The maximization step of the algorithm is carried out by two alternative methods, with one involving approximation using Taylor series expansion (leading to approximate maximum likelihood estimate) and the other based on the EM gradient algorithm (Lange, 1995). These two methods are compared based on Monte Carlo simulations. The Fisher scoring method for obtaining the maximum likelihood estimates shows a problem of convergence under this setup, except when the truncation percentage is small. The asymptotic variance-covariance matrix of the MLEs is derived by using the missing information principle (Louis, 1982), and then the asymptotic confidence intervals for scale and shape parameters are obtained and compared with corresponding bootstrap confidence intervals. Finally, some numerical examples are given to illustrate all the methods of inference developed here.     

Spatial sampling design for parameter estimation of the covariance function

We study the spatial optimal sampling design for covariance parameter estimation. The spatial process is modeled as a Gaussian random field and maximum likelihood (ML) is used to estimate the covariance parameters. We use the log determinant of the inverse Fisher information matrix as the design criterion and run simulations to investigate the relationship between the inverse Fisher information matrix and the covariance matrix of the ML estimates. A simulated annealing algorithm is developed to search for an optimal design among all possible designs on a fine grid. Since the design criterion depends on the unknown parameters, we define relative efficiency of a design and consider minimax and Bayesian criteria to find designs that are robust for a range of parameter values. Simulation results are presented for the Matérn class of covariance functions.     

Estimation on Lomax progressive censoring using the EM algorithm

Based on progressively type-II censored data, the maximum-likelihood estimators (MLEs) for the Lomax parameters are derived using the expectation–maximization (EM) algorithm. Moreover, the expected Fisher information matrix based on the missing value principle is computed. Using extensive simulation and three criteria, namely, bias, root mean squared error and Pitman closeness measures, we compare the performance of the MLEs via the EM algorithm and the Newton–Raphson (NR) method. It is concluded that the EM algorithm outperforms the NR method in all the cases. Two real data examples are used to illustrate our proposed estimators.     

Estimating the Burr XII Parameters in Constant-Stress Partially Accelerated Life Tests Under Multiple Censored Data

In this article, we present the performance of the maximum likelihood estimates of the Burr XII parameters for constant-stress partially accelerated life tests under multiple censored data. Two maximum likelihood estimation methods are considered. One method is based on observed-data likelihood function and the maximum likelihood estimates are obtained by using the quasi-Newton algorithm. The other method is based on complete-data likelihood function and the maximum likelihood estimates are derived by using the expectation-maximization (EM) algorithm. The variance–covariance matrices are derived to construct the confidence intervals of the parameters. The performance of these two algorithms is compared with each other by a simulation study. The simulation results show that the maximum likelihood estimation via the EM algorithm outperforms the quasi-Newton algorithm in terms of the absolute relative bias, the bias, the root mean square error and the coverage rate. Finally, a numerical example is given to illustrate the performance of the proposed methods.     

Estimation and prediction for a Burr type-III distribution with progressive censoring

We consider estimation of unknown parameters and reliability characteristics of a Burr type-III distribution under progressive censoring. Predictive estimates for censored observations and the associated prediction intervals are also obtained. We derive maximum-likelihood estimators of unknown quantities using the EM algorithm and then also obtain the observed Fisher information matrix. We provide various Bayes estimators for unknown parameters under the squared error loss function. Highest posterior density and asymptotic intervals are also constructed. We evaluate performance of proposed methods using simulations. Finally, an illustrative example is presented in support of the methods discussed.     

Bivariate Kumaraswamy distribution: properties and a new method to generate bivariate classes

In this paper, we introduce a bivariate Kumaraswamy (BVK) distribution whose marginals are Kumaraswamy distributions. The cumulative distribution function of this bivariate model has absolutely continuous and singular parts. Representations for the cumulative and density functions are presented and properties such as marginal and conditional distributions, product moments and conditional moments are obtained. We show that the BVK model can be obtained from the Marshall and Olkin survival copula and obtain a tail dependence measure. The estimation of the parameters by maximum likelihood is discussed and the Fisher information matrix is determined. We propose an EM algorithm to estimate the parameters. Some simulations are presented to verify the performance of the direct maximum-likelihood estimation and the proposed EM algorithm. We also present a method to generate bivariate distributions from our proposed BVK distribution. Furthermore, we introduce a BVK distribution which has only an absolutely continuous part and discuss some of its properties. Finally, a real data set is analysed for illustrative purposes.     

Maximum likelihood estimates in the multivariate normal with patterned mean and covariance via the em algorithm

The maximum likelihood equations for a multivariate normal model with structured mean and structured covariance matrix may not have an explicit solution. In some cases the model's error term may be decomposed as the sum of two independent error terms, each having a patterned covariance matrix, such that if one of the unobservable error terms is artificially treated as "missing data", the EM algorithm can be used to compute the maximum likelihood estimates for the original problem. Some decompositions produce likelihood equations which do not have an explicit solution at each iteration of the EM algorithm, but within-iteration explicit solutions are shown for two general classes of models including covariance component models used for analysis of longitudinal data.     

Parameter Estimation of Type-II Hybrid Censored Weighted Exponential Distribution

A hybrid censoring scheme is a mixture of Type-I and Type-II censoring schemes. We study the estimation of parameters of weighted exponential distribution based on Type-II hybrid censored data. By applying the EM algorithm, maximum likelihood estimators are evaluated. Using Fisher information matrix, asymptotic confidence intervals are provided. By applying Markov chain Monte Carlo techniques, Bayes estimators, and corresponding highest posterior density confidence intervals of parameters are obtained. Monte Carlo simulations are performed to compare the performances of the different methods, and one dataset is analyzed for illustrative purposes.     

The competing risks analysis for parallel and series systems using Type-II progressive censoring

Abstract

Recently, the study of the lifetime of systems in reliability and survival analysis in the presence of several causes of failure (competing risks) has attracted attention in the literature. In this paper, series and parallel systems with exponential lifetime for each item of the system are considered. Several causes of failure independently affect lifetime distributions and observations of failure times of the systems are considered under progressive Type-II censored scheme. For series systems, the maximum likelihood estimates of parameters are computed and confidence intervals for parameters of the model are obtained using Fisher information matrix. For parallel systems, the generalized EM algorithm which uses the Newton-Raphson algorithm inside the EM algorithm is used to compute the maximum likelihood estimates of parameters. Also, the standard errors of the maximum likelihood estimates are computed by using the supplemented EM algorithm. The simulation study confirms the good performance of the introduced approach.     

Strategies for Fitting Large,Geostatistical Data in MCMC Simulation

Models for geostatistical data introduce spatial dependence in the covariance matrix of location-specific random effects. This is usually defined to be a parametric function of the distances between locations. Bayesian formulations of such models overcome asymptotic inference and estimation problems involved in maximum likelihood-based approaches and can be fitted using Markov chain Monte Carlo (MCMC) simulation. The MCMC implementation, however, requires repeated inversions of the covariance matrix which makes the problem computationally intensive, especially for large number of locations. In the present work, we propose to convert the spatial covariance matrix to a sparse matrix and compare a number of numerical algorithms especially suited within the MCMC framework in order to accelerate large matrix inversion. The algorithms are assessed empirically on simulated datasets of different size and sparsity. We conclude that the band solver applied after ordering the distance matrix reduces the computational time in inverting covariance matrices substantially.     

A cautionary note on generalized linear models for covariance of unbalanced longitudinal data

Missing data in longitudinal studies can create enormous challenges in data analysis when coupled with the positive-definiteness constraint on a covariance matrix. For complete balanced data, the Cholesky decomposition of a covariance matrix makes it possible to remove the positive-definiteness constraint and use a generalized linear model setup to jointly model the mean and covariance using covariates (Pourahmadi, 2000). However, this approach may not be directly applicable when the longitudinal data are unbalanced, as coherent regression models for the dependence across all times and subjects may not exist. Within the existing generalized linear model framework, we show how to overcome this and other challenges by embedding the covariance matrix of the observed data for each subject in a larger covariance matrix and employing the familiar EM algorithm to compute the maximum likelihood estimates of the parameters and their standard errors. We illustrate and assess the methodology using real data sets and simulations.     

Trajectory Modeling of Longitudinal Binary Data: Application of the EM Algorithm for Mixture Models

A developmental trajectory describes the course of behavior over time. Identifying multiple trajectories within an overall developmental process permits a focus on subgroups of particular interest. We introduce a framework for identifying trajectories by using the Expectation-Maximization (EM) algorithm to fit semiparametric mixtures of logistic distributions to longitudinal binary data. For performance comparison, we consider full maximization algorithms (PROC TRAJ in SAS), standard EM, and two other EM-based algorithms for speeding up convergence. Simulation shows that EM methods produce more accurate parameter estimates. The EM methodology is illustrated with a longitudinal dataset involving adolescents smoking behaviors.