An improved SAEM algorithm for maximum likelihood estimation in mixtures of non linear mixed effects models

We propose a new methodology for maximum likelihood estimation in mixtures of non linear mixed effects models (NLMEM). Such mixtures of models include mixtures of distributions, mixtures of structural models and mixtures of residual error models. Since the individual parameters inside the NLMEM are not observed, we propose to combine the EM algorithm usually used for mixtures models when the mixture structure concerns an observed variable, with the Stochastic Approximation EM (SAEM) algorithm, which is known to be suitable for maximum likelihood estimation in NLMEM and also has nice theoretical properties. The main advantage of this hybrid procedure is to avoid a simulation step of unknown group labels required by a “full” version of SAEM. The resulting MSAEM (Mixture SAEM) algorithm is now implemented in the Monolix software. Several criteria for classification of subjects and estimation of individual parameters are also proposed. Numerical experiments on simulated data show that MSAEM performs well in a general framework of mixtures of NLMEM. Indeed, MSAEM provides an estimator close to the maximum likelihood estimator in very few iterations and is robust with regard to initialization. An application to pharmacokinetic (PK) data demonstrates the potential of the method for practical applications.     

Maximum likelihood from spatial random effects models via the stochastic approximation expectation maximization algorithm

We introduce a class of spatial random effects models that have Markov random fields (MRF) as latent processes. Calculating the maximum likelihood estimates of unknown parameters in SREs is extremely difficult, because the normalizing factors of MRFs and additional integrations from unobserved random effects are computationally prohibitive. We propose a stochastic approximation expectation-maximization (SAEM) algorithm to maximize the likelihood functions of spatial random effects models. The SAEM algorithm integrates recent improvements in stochastic approximation algorithms; it also includes components of the Newton-Raphson algorithm and the expectation-maximization (EM) gradient algorithm. The convergence of the SAEM algorithm is guaranteed under some mild conditions. We apply the SAEM algorithm to three examples that are representative of real-world applications: a state space model, a noisy Ising model, and segmenting magnetic resonance images (MRI) of the human brain. The SAEM algorithm gives satisfactory results in finding the maximum likelihood estimate of spatial random effects models in each of these instances.     

A simulated annealing version of the EM algorithm for non-Gaussian deconvolution

The Expectation–Maximization (EM) algorithm is a very popular technique for maximum likelihood estimation in incomplete data models. When the expectation step cannot be performed in closed form, a stochastic approximation of EM (SAEM) can be used. Under very general conditions, the authors have shown that the attractive stationary points of the SAEM algorithm correspond to the global and local maxima of the observed likelihood. In order to avoid convergence towards a local maxima, a simulated annealing version of SAEM is proposed. An illustrative application to the convolution model for estimating the coefficients of the filter is given.     

Understanding and Addressing the Unbounded “Likelihood” Problem

The joint probability density function, evaluated at the observed data, is commonly used as the likelihood function to compute maximum likelihood estimates. For some models, however, there exist paths in the parameter space along which this density-approximation likelihood goes to infinity and maximum likelihood estimation breaks down. In all applications, however, observed data are really discrete due to the round-off or grouping error of measurements. The “correct likelihood” based on interval censoring can eliminate the problem of an unbounded likelihood. This article categorizes the models leading to unbounded likelihoods into three groups and illustrates the density-approximation breakdown with specific examples. Although it is usually possible to infer how given data were rounded, when this is not possible, one must choose the width for interval censoring, so we study the effect of the round-off on estimation. We also give sufficient conditions for the joint density to provide the same maximum likelihood estimate as the correct likelihood, as the round-off error goes to zero.     

A parameter expansion version of the SAEM algorithm

The EM algorithm and its extensions are very popular tools for maximum likelihood estimation in incomplete data setting. However, one of the limitations of these methods is their slow convergence. The PX-EM (parameter-expanded EM) algorithm was proposed by Liu, Rubin and Wu to make EM much faster. On the other hand, stochastic versions of EM are powerful alternatives of EM when the E-step is untractable in a closed form. In this paper we propose the PX-SAEM which is a parameter expansion version of the so-called SAEM (Stochastic Approximation version of EM). PX-SAEM is shown to accelerate SAEM and improve convergence toward the maximum likelihood estimate in a parametric framework. Numerical examples illustrate the behavior of PX-SAEM in linear and nonlinear mixed effects models.     

Variational Bayes with synthetic likelihood

Synthetic likelihood is an attractive approach to likelihood-free inference when an approximately Gaussian summary statistic for the data, informative for inference about the parameters, is available. The synthetic likelihood method derives an approximate likelihood function from a plug-in normal density estimate for the summary statistic, with plug-in mean and covariance matrix obtained by Monte Carlo simulation from the model. In this article, we develop alternatives to Markov chain Monte Carlo implementations of Bayesian synthetic likelihoods with reduced computational overheads. Our approach uses stochastic gradient variational inference methods for posterior approximation in the synthetic likelihood context, employing unbiased estimates of the log likelihood. We compare the new method with a related likelihood-free variational inference technique in the literature, while at the same time improving the implementation of that approach in a number of ways. These new algorithms are feasible to implement in situations which are challenging for conventional approximate Bayesian computation methods, in terms of the dimensionality of the parameter and summary statistic.     

Modified profile likelihood estimation for the Weibull regression models in survival analysis

In this study, adjustment of profile likelihood function of parameter of interest in presence of many nuisance parameters is investigated for survival regression models. Our objective is to extend the Barndorff–Nielsen’s technique to Weibull regression models for estimation of shape parameter in presence of many nuisance and regression parameters. We conducted Monte-Carlo simulation studies and a real data analysis, all of which demonstrate and suggest that the modified profile likelihood estimators outperform the profile likelihood estimators in terms of three comparison criterion: mean squared errors, bias and standard errors.     

Parameter estimation in semi-linear models using a maximal invariant likelihood function

In this paper, we consider the problem of estimation of semi-linear regression models. Using invariance arguments, Bhowmik and King [2007. Maximal invariant likelihood based testing of semi-linear models. Statist. Papers 48, 357–383] derived the probability density function of the maximal invariant statistic for the non-linear component of these models. Using this density function as a likelihood function allows us to estimate these models in a two-step process. First the non-linear component parameters are estimated by maximising the maximal invariant likelihood function. Then the non-linear component, with the parameter values replaced by estimates, is treated as a regressor and ordinary least squares is used to estimate the remaining parameters. We report the results of a simulation study conducted to compare the accuracy of this approach with full maximum likelihood and maximum profile-marginal likelihood estimation. We find maximising the maximal invariant likelihood function typically results in less biased and lower variance estimates than those from full maximum likelihood.     

Generalized empirical likelihood inference in partial linear regression model for longitudinal data

In this paper, empirical likelihood inference for longitudinal data within the framework of partial linear regression models are investigated. The proposed procedures take into consideration the correlation within groups without involving direct estimation of nuisance parameters in the correlation matrix. The empirical likelihood method is used to estimate the regression coefficients and the baseline function, and to construct confidence intervals. A nonparametric version of Wilk's theorem for the limiting distribution of the empirical likelihood ratio is derived. Compared with methods based on normal approximations, the empirical likelihood does not require consistent estimators for the asymptotic variance and bias. The finite sample behaviour of the proposed method is evaluated with simulation and illustrated with an AIDS clinical trial data set.     

Empirical Likelihood Inferences for Semiparametric Varying-Coefficient Partially Linear Models with Longitudinal Data

In this article, empirical likelihood inferences for semiparametric varying-coefficient partially linear models with longitudinal data are investigated. We propose a groupwise empirical likelihood procedure to handle the inter-series dependence of the longitudinal data. By using residual-adjustment, an empirical likelihood ratio function for the nonparametric component is constructed, and a nonparametric version Wilks' phenomenons is proved. Compared with methods based on normal approximations, the empirical likelihood does not require consistent estimators for the asymptotic variance and bias. A simulation study is undertaken to assess the finite sample performance of the proposed confidence regions.     

Learning from a lot: Empirical Bayes for high‐dimensional model‐based prediction

Empirical Bayes is a versatile approach to “learn from a lot” in two ways: first, from a large number of variables and, second, from a potentially large amount of prior information, for example, stored in public repositories. We review applications of a variety of empirical Bayes methods to several well‐known model‐based prediction methods, including penalized regression, linear discriminant analysis, and Bayesian models with sparse or dense priors. We discuss “formal” empirical Bayes methods that maximize the marginal likelihood but also more informal approaches based on other data summaries. We contrast empirical Bayes to cross‐validation and full Bayes and discuss hybrid approaches. To study the relation between the quality of an empirical Bayes estimator and p, the number of variables, we consider a simple empirical Bayes estimator in a linear model setting. We argue that empirical Bayes is particularly useful when the prior contains multiple parameters, which model a priori information on variables termed “co‐data”. In particular, we present two novel examples that allow for co‐data: first, a Bayesian spike‐and‐slab setting that facilitates inclusion of multiple co‐data sources and types and, second, a hybrid empirical Bayes–full Bayes ridge regression approach for estimation of the posterior predictive interval.     

Analytic bias correction for maximum likelihood estimators when the bias function is non-constant

Recently, many articles have obtained analytical expressions for the biases of various maximum likelihood estimators, despite their lack of closed-form solution. These bias expressions have provided an attractive alternative to the bootstrap. Unless the bias function is “flat,” however, the expressions are being evaluated at the wrong point(s). We propose an “improved” analytical bias-adjusted estimator, in which the bias expression is evaluated at a more appropriate point (at the bias adjusted estimator itself). Simulations illustrate that the improved analytical bias-adjusted estimator can eliminate significantly more bias than the simple estimator, which has been well established in the literature.     

On the small sample properties of norm-restricted maximum likelihood estimators for logistic regression models

This paper develops alternatives to maximum likelihood estimators (MLE) for logistic regression models and compares the mean squared error (MSE) of the estimators. The MLE for the vector of underlying success probabilities has low MSE only when the true probabilities are extreme (i.e., near 0 or 1). Extreme probabilities correspond to logistic regression parameter vectors which are large in norm. A competing “restricted” MLE and an empirical version of it are suggested as estimators with better performance than the MLE for central probabilities. An approximate EM-algorithm for estimating the restriction is described. As in the case of normal theory ridge estimators, the proposed estimators are shown to be formally derivable by Bayes and empirical Bayes arguments. The small sample operating characteristics of the proposed estimators are compared to the MLE via a simulation study; both the estimation of individual probabilities and of logistic parameters are considered.     

GMM Estimation of Count-Panel-Data Models With Fixed Effects and Predetermined Instruments

The “traditional” approach to the estimation of count-panel-data models with fixed effects is the conditional maximum likelihood estimator. The pseudo maximum likelihood principle can be used in these models to obtain orthogonality conditions that generate a robust estimator. This estimator is inconsistent, however, when the instruments are not strictly exogenous. This article proposes a generalized method of moments estimator for count-panel-data models with fixed effects, based on a transformation of the conditional mean specification, that is consistent even when the explanatory variables are predetermined. Two applications are discussed, the relationship between patents and research and development expenditures and the explanation of technology transfer.     

Maximum likelihood estimators for ARMA and ARFIMA models: a Monte Carlo study

We analyze by simulation the properties of two time domain and two frequency domain estimators for low-order autoregressive fractionally integrated moving-average Gaussian models, ARFIMA (p,d,q). The estimators considered are the exact maximum likelihood for demeaned data (EML) the associated modified profile likelihood (MPL) and the Whittle estimator with (WLT) and without tapered data (WL). Length of the series is 100. The estimators are compared in terms of pile-up effect, mean square error, bias, and empirical confidence level. The tapered version of the Whittle likelihood turns out to be a reliable estimator for ARMA and ARFIMA models. Its small losses in performance in case of ‘well-behaved’ models are compensated sufficiently in more ‘difficult’ models. The modified profile likelihood is an alternative to the WLT but is computationally more demanding. It is either equivalent to the EML or more favorable than the EML. For fractionally integrated models, particularly, it dominates clearly the EML. The WL has serious deficiencies for large ranges of parameters, and so cannot be recommended in general. The EML, on the other hand, should only be used with care for fractionally integrated models due to its potential large negative bias of the fractional integration parameter. In general, one should proceed with caution for ARMA(1,1) models with almost canceling roots, and, in particular, in case of the EML and the MPL for inference in the vicinity of a moving-average root of +1.     

ESTIMATION OF FAMILIAL CORRELATIONS UNDER AUTOREGRESSIVE CIRCULAR COVARIANCE

Circular covariance matrices play an important role in modeling phenomena in numerous epidemiological, communications and physical contexts. In this article, we propose a parsimonious, autoregressive type of circular covariance structure for modeling correlations between the “siblings” of a “family”. This structure, similar to AR(1) structure used in time series models, involves only two parameters. We derive the maximum likelihood estimators of these parameters, and discuss testing of hypotheses about the autoregressive parameter. Estimation of “parent-sib” correlation, namely, the interclass correlation, is also considered. Estimation of the parameters when there are unequal numbers of siblings in different families is also discussed.     

Likelihood-free inference via classification

Increasingly complex generative models are being used across disciplines as they allow for realistic characterization of data, but a common difficulty with them is the prohibitively large computational cost to evaluate the likelihood function and thus to perform likelihood-based statistical inference. A likelihood-free inference framework has emerged where the parameters are identified by finding values that yield simulated data resembling the observed data. While widely applicable, a major difficulty in this framework is how to measure the discrepancy between the simulated and observed data. Transforming the original problem into a problem of classifying the data into simulated versus observed, we find that classification accuracy can be used to assess the discrepancy. The complete arsenal of classification methods becomes thereby available for inference of intractable generative models. We validate our approach using theory and simulations for both point estimation and Bayesian inference, and demonstrate its use on real data by inferring an individual-based epidemiological model for bacterial infections in child care centers.     

Extending conditional likelihood in models for stratified binary data

The conditional likelihood is widely used in logistic regression models with stratified binary data. In particular, it leads to accurate inference for the parameters of interest, which are common to all strata, eliminating stratum-specific nuisance parameters. The modified profile likelihood is an accurate approximation to the conditional likelihood, but has the advantage of being available for general parametric models. Here, we propose the modified profile likelihood as an ideal extension of the conditional likelihood in generalized linear models for binary data, with generic link function. An important feature is that for the implementation we only need standard outputs of routines for generalized linear models. The accuracy of the method is supported by theoretical properties and is confirmed by simulation results.This research was supported by MIUR COFIN 2001-2003.     

Bayesian Analysis in Regression Models Using Pseudo-Likelihoods

This article deals with the issue of using a suitable pseudo-likelihood, instead of an integrated likelihood, when performing Bayesian inference about a scalar parameter of interest in the presence of nuisance parameters. The proposed approach has the advantages of avoiding the elicitation on the nuisance parameters and the computation of multidimensional integrals. Moreover, it is particularly useful when it is difficult, or even impractical, to write the full likelihood function.

We focus on Bayesian inference about a scalar regression coefficient in various regression models. First, in the context of non-normal regression-scale models, we give a theroetical result showing that there is no loss of information about the parameter of interest when using a posterior distribution derived from a pseudo-likelihood instead of the correct posterior distribution. Second, we present non trivial applications with high-dimensional, or even infinite-dimensional, nuisance parameters in the context of nonlinear normal heteroscedastic regression models, and of models for binary outcomes and count data, accounting also for possibile overdispersion. In all these situtations, we show that non Bayesian methods for eliminating nuisance parameters can be usefully incorporated into a one-parameter Bayesian analysis.     

Consistency of semiparametric maximum likelihood estimators for two‐phase sampling

Semiparametric maximum likelihood estimators have recently been proposed for a class of two‐phase, outcome‐dependent sampling models. All of them were “restricted” maximum likelihood estimators, in the sense that the maximization is carried out only over distributions concentrated on the observed values of the covariate vectors. In this paper, the authors give conditions for consistency of these restricted maximum likelihood estimators. They also consider the corresponding unrestricted maximization problems, in which the “absolute” maximum likelihood estimators may then have support on additional points in the covariate space. Their main consistency result also covers these unrestricted maximum likelihood estimators, when they exist for all sample sizes.