Likelihood estimation of missing cell means in the fixed model analysis of variance

This paper examines the formation of maximum likelihood estimates of cell means in analysis of variance problems for cells with missing observations. Methods of estimating the means for missing cells has a long history which includes iterative maximum likelihood techniques, approximation techniques and ad hoc techniques. The use of the EM algorithm to form maximum likelihood estimates has resolved most of the issues associated with this problem. Implementation of the EM algorithm entails specification of a reduced model. As demonstrated in this paper, when there are several missing cells, it is possible to specify a reduced model that results in an unidentifiable likelihood. The EM algorithm in this case does not converge, although the slow divergence may often be mistaken by the unwary as convergence. This paper presents a simple matrix method of determining whether or not the reduced model results in an identifiable likelihood, and consequently in an EM algorithm that converges. We also show the EM algorithm in this case to be equivalent to a method which yields a closed form solution.     

A likelihood ratio test of quasi-independence for sparse two-way contingency tables

We consider a likelihood ratio test of independence for large two-way contingency tables having both structural (non-random) and sampling (random) zeros in many cells. The solution of this problem is not available using standard likelihood ratio tests. One way to bypass this problem is to remove the structural zeroes from the table and implement a test on the remaining cells which incorporate the randomness in the sampling zeros; the resulting test is a test of quasi-independence of the two categorical variables. This test is based only on the positive counts in the contingency table and is valid when there is at least one sampling (random) zero. The proposed (likelihood ratio) test is an alternative to the commonly used ad hoc procedures of converting the zero cells to positive ones by adding a small constant. One practical advantage of our procedure is that there is no need to know if a zero cell is structural zero or a sampling zero. We model the positive counts using a truncated multinomial distribution. In fact, we have two truncated multinomial distributions; one for the null hypothesis of independence and the other for the unrestricted parameter space. We use Monte Carlo methods to obtain the maximum likelihood estimators of the parameters and also the p-value of our proposed test. To obtain the sampling distribution of the likelihood ratio test statistic, we use bootstrap methods. We discuss many examples, and also empirically compare the power function of the likelihood ratio test relative to those of some well-known test statistics.     

AN ALTERNATIVE PARAMETRIC APPROACH FOR DISCRETE MISSING DATA PROBLEMS

We propose an iterative method of estimation for discrete missing data problems that is conceptually different from the Expectation–Maximization (EM) algorithm and that does not in general yield the observed data maximum likelihood estimate (MLE). The proposed approach is based conceptually upon weighting the set of possible complete-data MLEs. Its implementation avoids the expectation step of EM, which can sometimes be problematic. In the simple case of Bernoulli trials missing completely at random, the iterations of the proposed algorithm are equivalent to the EM iterations. For a familiar genetics-oriented multinomial problem with missing count data and for the motivating example with epidemiologic applications that involves a mixture of a left censored normal distribution with a point mass at zero, we investigate the finite sample performance of the proposed estimator and find it to be competitive with that of the MLE. We give some intuitive justification for the method, and we explore an interesting connection between our algorithm and multiple imputation in order to suggest an approach for estimating standard errors.     

Maximum likelihood estimation for incomplete multinomial data via the weaver algorithm

In a multinomial model, the sample space is partitioned into a disjoint union of cells. The partition is usually immutable during sampling of the cell counts. In this paper, we extend the multinomial model to the incomplete multinomial model by relaxing the constant partition assumption to allow the cells to be variable and the counts collected from non-disjoint cells to be modeled in an integrated manner for inference on the common underlying probability. The incomplete multinomial likelihood is parameterized by the complete-cell probabilities from the most refined partition. Its sufficient statistics include the variable-cell formation observed as an indicator matrix and all cell counts. With externally imposed structures on the cell formation process, it reduces to special models including the Bradley–Terry model, the Plackett–Luce model, etc. Since the conventional method, which solves for the zeros of the score functions, is unfruitful, we develop a new approach to establishing a simpler set of estimating equations to obtain the maximum likelihood estimate (MLE), which seeks the simultaneous maximization of all multiplicative components of the likelihood by fitting each component into an inequality. As a consequence, our estimation amounts to solving a system of the equality attainment conditions to the inequalities. The resultant MLE equations are simple and immediately invite a fixed-point iteration algorithm for solution, which is referred to as the weaver algorithm. The weaver algorithm is short and amenable to parallel implementation. We also derive the asymptotic covariance of the MLE, verify main results with simulations, and compare the weaver algorithm with an MM/EM algorithm based on fitting a Plackett–Luce model to a benchmark data set.     

Likelihood estimation for exchangeable multinomial data

In this article, maximum likelihood estimates of an exchangeable multinomial distribution using a parametric form to model the parameters as functions of covariates are derived. The non linearity of the exchangeable multinomial distribution and the parametric model make direct application of Newton Rahpson and Fisher's scoring algorithms computationally infeasible. Instead parameter estimates are obtained as solutions to an iterative weighted least-squares algorithm. A completely monotonic parametric form is proposed for defining the marginal probabilities that results in a valid probability model.     

Parameter estimation from interval-valued data using the expectation-maximization algorithm

This paper investigates on the problem of parameter estimation in statistical model when observations are intervals assumed to be related to underlying crisp realizations of a random sample. The proposed approach relies on the extension of likelihood function in interval setting. A maximum likelihood estimate of the parameter of interest may then be defined as a crisp value maximizing the generalized likelihood function. Using the expectation-maximization (EM) to solve such maximizing problem therefore derives the so-called interval-valued EM algorithm (IEM), which makes it possible to solve a wide range of statistical problems involving interval-valued data. To show the performance of IEM, the following two classical problems are illustrated: univariate normal mean and variance estimation from interval-valued samples, and multiple linear/nonlinear regression with crisp inputs and interval output.     

Numerical estimation of a probability measure

An iterative solution to the problem of maximizing a concave functional ø defined on the set of all probability measures on a topological space is considered. Convergence of this procedure and a rapidly converging algorithm are studied. Computational aspects of this algorithm along with the ones developed earlier by Wynn, Fedorov, Atwood, Wu and others are provided. Examples discussed are taken from the area of mixture likehoods and optimal experimental design.     

A self-consistency approach to multinomial logit model with random effects

The computation in the multinomial logit mixed effects model is costly especially when the response variable has a large number of categories, since it involves high-dimensional integration and maximization. Tsodikov and Chefo (2008) developed a stable MLE approach to problems with independent observations, based on generalized self-consistency and quasi-EM algorithm developed in Tsodikov (2003). In this paper, we apply the idea to clustered multinomial response to simplify the maximization step. The method transforms the complex multinomial likelihood to Poisson-type likelihood and hence allows for the estimates to be obtained iteratively solving a set of independent low-dimensional problems. The methodology is applied to real data and studied by simulations. While maximization is simplified, numerical integration remains the dominant challenge to computational efficiency.     

An alternative algorithm of the empirical likelihood estimation for the parameter of a linear regression model

ABSTRACT

Empirical likelihood (EL) is a nonparametric method based on observations. EL method is defined as a constrained optimization problem. The solution of this constrained optimization problem is carried on using duality approach. In this study, we propose an alternative algorithm to solve this constrained optimization problem. The new algorithm is based on a newton-type algorithm for Lagrange multipliers for the constrained optimization problem. We provide a simulation study and a real data example to compare the performance of the proposed algorithm with the classical algorithm. Simulation and the real data results show that the performance of the proposed algorithm is comparable with the performance of the existing algorithm in terms of efficiencies and cpu-times.     

Changepoint Analysis as a Method for Isotonic Inference

Concavity and sigmoidicity hypotheses are developed as a natural extension of the simple ordered hypothesis in normal means. Those hypotheses give reasonable shape constraints for obtaining a smooth response curve in the non-parametric inputoutput analysis. The slope change and inflection point models are introduced correspondingly as the corners of the polyhedral cones defined by those isotonic hypotheses. Then a maximal contrast type test is derived systematically as the likelihood ratio test for each of those changepoint hypotheses. The test is also justified for the original isotonic hypothesis by a complete class lemma. The component variables of the resulting test statistic have second or third order Markov property which, together with an appropriate non-linear transformation, leads to an exact and very efficient algorithm for the probability calculation. Some considerations on the power of the test are given showing this to be a very promising way of approaching to the isotonic inference.     

Domains of convergence for the EM algorithm: a cautionary tale in a location estimation problem

The EM algorithm is a popular method for maximizing a likelihood in the presence of incomplete data. When the likelihood has multiple local maxima, the parameter space can be partitioned into domains of convergence, one for each local maximum. In this paper we investigate these domains for the location family generated by the t-distribution. We show that, perhaps somewhat surprisingly, these domains need not be connected sets. As an extreme case we give an example of a domain which consists of an infinite union of disjoint open intervals. Thus the convergence behaviour of the EM algorithm can be quite sensitive to the starting point.     

Generalized Self-Consistency: Multinomial logit model and Poisson likelihood

A generalized self-consistency approach to maximum likelihood estimation (MLE) and model building was developed in Tsodikov [2003. Semiparametric models: a generalized self-consistency approach. J. Roy. Statist. Soc. Ser. B Statist. Methodology 65(3), 759–774] and applied to a survival analysis problem. We extend the framework to obtain second-order results such as information matrix and properties of the variance. Multinomial model motivates the paper and is used throughout as an example. Computational challenges with the multinomial likelihood motivated Baker [1994. The Multinomial–Poisson transformation. The Statist. 43, 495–504] to develop the Multinomial–Poisson (MP) transformation for a large variety of regression models with multinomial likelihood kernel. Multinomial regression is transformed into a Poisson regression at the cost of augmenting model parameters and restricting the problem to discrete covariates. Imposing normalization restrictions by means of Lagrange multipliers [Lang, J., 1996. On the comparison of multinomial and Poisson log-linear models. J. Roy. Statist. Soc. Ser. B Statist. Methodology 58, 253–266] justifies the approach. Using the self-consistency framework we develop an alternative solution to multinomial model fitting that does not require augmenting parameters while allowing for a Poisson likelihood and arbitrary covariate structures. Normalization restrictions are imposed by averaging over artificial “missing data” (fake mixture). Lack of probabilistic interpretation at the “complete-data” level makes the use of the generalized self-consistency machinery essential.     

S–S method for stochastically ordered multinomial populations with missing data

Stochastic ordering is a useful concept in order restricted inferences. In this paper, we propose a new estimation technique for the parameters in two multinomial populations under stochastic orderings when missing data are present. In comparison with traditional maximum likelihood estimation method, our new method can guarantee the uniqueness of the maximum of the likelihood function. Furthermore, it does not depend on the choice of initial values for the parameters in contrast to the EM algorithm. Finally, we give the asymptotic distributions of the likelihood ratio statistics based on the new estimation method.     

Experiments with maximum likelihood method for image motion deblurring

Relative motion between the camera and the object results in the recording of a motion-blurred image. Under certain idealized conditions, such blurring can be mathematically corrected. We refer to this as 'motion deblurring'. We start with some idealized assumptions under which the motion deblurring problem is a linear inverse problem with certain positivity constraints; LININPOS problems, for short. Such problems, even in the case of no statistical noise, can be solved using the maximum likelihood/EM approach in the following sense. If they have a solution, the ML/EM iterative method will converge to it; otherwise, it will converge to the nearest approximation of a solution, where 'nearest' is interpreted in a likelihood sense or, equivalently, in a Kullback-Leibler information divergence sense. We apply the ML/EM algorithm to such problems and discuss certain special cases, such as motion along linear or circular paths with or without acceleration. The idealized assumptions under which the method is developed are hardly ever satisfied in real applications, so we experiment with the method under conditions that violate these assumptions. Specifically, we experimented with an image created through a computer-simulated digital motion blurring corrupted with noise, and with an image of a moving toy cart recorded with a 35 mm camera while in motion. The gross violations of the idealized assumptions, especially in the toy cart example, led to a host of very difficult problems which always occur under real-life conditions and need to be addressed. We discuss these problems in detail and propose some 'engineering solutions' that, when put together, appear to lead to a good methodology for certain motion deblurring problems. Some of the issues we discuss, in various degrees of detail, include estimating the speed of motion which is referred to as 'blur identification'; non-zero-background artefacts and pre- and post- processing of the images to remove such artefacts; the need to 'stabilize' the solution because of the inherent ill-posedness of the problem; and computer implemetation.     

Experiments with maximum likelihood method for image motion deblurring

Relative motion between the camera and the object results in the recording of a motion-blurred image. Under certain idealized conditions, such blurring can be mathematically corrected. We refer to this as ‘motion deblurring’. We start with some idealized assumptions under which the motion deblurring problem is a linear inverse problem with certain positivity constraints; LININPOS problems, for short. Such problems, even in the case of no statistical noise, can be solved using the maximum likelihood/EM approach in the following sense. If they have a solution, the ML/EM iterative method will converge to it; otherwise, it will converge to the nearest approximation of a solution, where ‘nearest’ is interpreted in a likelihood sense or, equivalently, in a Kullback-Leibler information divergence sense. We apply the ML/EM algorithm to such problems and discuss certain special cases, such as motion along linear or circular paths with or without acceleration. The idealized assumptions under which the method is developed are hardly ever satisfied in real applications, so we experiment with the method under conditions that violate these assumptions. Specifically, we experimented with an image created through a computer-simulated digital motion blurring corrupted with noise, and with an image of a moving toy cart recorded with a 35 mm camera while in motion. The gross violations of the idealized assumptions, especially in the toy cart example, led to a host of very difficult problems which always occur under real-life conditions and need to be addressed. We discuss these problems in detail and propose some ‘engineering solutions' that, when put together, appear to lead to a good methodology for certain motion deblurring problems. Some of the issues we discuss, in various degrees of detail, include estimating the speed of motion which is referred to as ‘blur identification’; non-zero-background artefacts and pre- and post- processing of the images to remove such artefacts; the need to ‘stabilize’ the solution because of the inherent ill-posedness of the problem; and computer implemetation.     

Estimation of two order restricted normal means with unknown and possibly unequal variances

Estimation of each of and linear functions of two order restricted normal means is considered when variances are unknown and possibly unequal. We replace unknown variances with sample variances and construct isotonic regression estimators, which we call in our paper the plug-in estimators, to estimate ordered normal means. Under squared error loss, a necessary and sufficient condition is given for the plug-in estimators to improve upon the unrestricted maximum likelihood estimators uniformly. As for the estimation of linear functions of ordered normal means, we also show that when variances are known, the restricted maximum likelihood estimator always improves upon the unrestricted maximum likelihood estimator uniformly, but when variances are unknown, the plug-in estimator does not always improve upon the unrestricted maximum likelihood estimator uniformly.     

Likelihood ratio test against simple stochastic ordering among several multinomial populations

Stochastic ordering between probability distributions has been widely studied in the past 50 years. Because it is often easy to make valuable judgments when such orderings exist, it is desirable to recognize their existence and to model distributional structures under them. Likelihood ratio test is the most commonly used method to test hypotheses involving stochastic orderings. Among the various formally defined notions of stochastic ordering, the least stringent is simple stochastic ordering. In this paper, we consider testing the hypothesis that all multinomial populations are identically distributed against the alternative that they are in simple stochastic ordering. We construct likelihood ratio test statistic for this hypothesis test problem, provide limit form of the objective function corresponding to the test statistic and show that the test statistic is asymptotically distributed as a mixture of chi-squared distributions, i.e., a chi-bar-squared distribution.     

Non parametric hazard estimation with dependent censoring using penalized likelihood and an assumed copula

This article introduces a novel non parametric penalized likelihood hazard estimation when the censoring time is dependent on the failure time for each subject under observation. More specifically, we model this dependence using a copula, and the method of maximum penalized likelihood (MPL) is adopted to estimate the hazard function. We do not consider covariates in this article. The non negatively constrained MPL hazard estimation is obtained using a multiplicative iterative algorithm. The consistency results and the asymptotic properties of the proposed hazard estimator are derived. The simulation studies show that our MPL estimator under dependent censoring with an assumed copula model provides a better accuracy than the MPL estimator under independent censoring if the sign of dependence is correctly specified in the copula function. The proposed method is applied to a real dataset, with a sensitivity analysis performed over various values of correlation between failure and censoring times.     

A model with mixed binary responses and censored observations

In this paper, we study the maximum likelihood estimation of a model with mixed binary responses and censored observations. The model is very general and includes the Tobit model and the binary choice model as special cases. We show that, by using additional binary choice observations, our method is more efficient than the traditional Tobit model. Two iterative procedures are proposed to compute the maximum likelihood estimator (MLE) for the model based on the EM algorithm (Dempster et al, 1977) and the Newton-Raphson method. The uniqueness of the MLE is proved. The simulation results show that the inconsistency and inefficiency can be significant when the Tobit method is applied to the present mixed model. The experiment results also suggest that the EM algorithm is much faster than the Newton-Raphson method for the present mixed model. The method also allows one to combine two data sets, the smaller data set with more detailed observations and the larger data set with less detailed binary choice observations in order to improve the efficiency of estimation. This may entail substantial savings when one conducts surveys.     

Upper Bound for p-Value of the Test of Multivariate Normal Ordered Mean Vectors Against all Alternatives

Consider the problem of testing the isotonic of several p-variate normal mean vectors against all alternatives. It is difficult to compute the exact p-value for this problem of testing with the classical method when the covariance matrices are completely unknown. In the present paper, a test statistic is proposed for this problem of testing. A reformulation of the test statistic is given based on the orthogonal projections on the closed convex cones and then the upper bound for p-value of the test statistic is computed.