The nonparametric maximum likelihood estimator for middle-censored data

In this note, we consider data subjected to middle censoring where the variable of interest becomes unobservable when it falls within an interval of censorship. We demonstrate that the nonparametric maximum likelihood estimator (NPMLE) of distribution function can be obtained by using Turnbull's (1976) EM algorithm or self-consistent estimating equation (Jammalamadaka and Mangalam, 2003) with an initial estimator which puts mass only on the innermost intervals. The consistency of the NPMLE can be established based on the asymptotic properties of self-consistent estimators (SCE) with mixed interval-censored data ( [Yu et al., 2000] and [Yu et al., 2001]).     

Identity for the NPMLE in Censored Data Models

We derive an identity for nonparametric maximum likelihood estimators (NPMLE) and regularized MLEs in censored data models which expresses the standardized maximum likelihood estimator in terms of the standardized empirical process. This identity provides an effective starting point in proving both consistency and efficiency of NPMLE and regularized MLE. The identity and corresponding method for proving efficiency is illustrated for the NPMLE in the univariate right-censored data model, the regularized MLE in the current status data model and for an implicit NPMLE based on a mixture of right-censored and current status data. Furthermore, a general algorithm for estimation of the limiting variance of the NPMLE is provided. This revised version was published online in July 2006 with corrections to the Cover Date.     

Isotonic estimation of survival under a misattribution of cause of death

Several authors have indicated that incorrectly classified cause of death for prostate cancer survivors may have played a role in the observed recent peak and decline of prostate cancer mortality. Motivated by the suggestion we studied a competing risks model where other cause of death may be misattributed as a death of interest. We first consider a na?ve approach using unconstrained nonparametric maximum likelihood estimation (NPMLE), and then present the constrained NPMLE where the survival function is forced to be monotonic. Surprising observations were made as we studied their small-sample and asymptotic properties in continuous and discrete situations. Contrary to the common belief that the non-monotonicity of a survival function NPMLE is a small-sample problem, the constrained NPMLE is asymptotically biased in the continuous setting. Other isotonic approaches, the supremum (SUP) method and the Pooled-Adjacent-Violators (PAV) algorithm, and the EM algorithm are also considered. We found that the EM algorithm is equivalent to the constrained NPMLE. Both SUP method and PAV algorithm deliver consistent and asymptotically unbiased estimator. All methods behave well asymptotically in the discrete time setting. Data from the Surveillance, Epidemiology and End Results (SEER) database are used to illustrate the proposed estimators.     

The Two-interval Line-segment Problem

In this paper we define and study the non-parametric maximum likelihood estimator (NPMLE) in the one-dimensional line-segment problem, where we observe line-segments on the real line through an interval with a gap which is smaller than the two remaining intervals. We define the self-consistency equations for the NPMLE and provide a quick algorithm for solving them. We prove supremum norm weak convergence to a Gaussian process and efficiency of the NPMLE. The problem has a geological application in the study of the lifespan of species     

A Note on Inconsistency of NPMLE of the Distribution Function from Left Truncated and Case I Interval Censored Data

We show that under reasonable conditions the nonparametric maximum likelihood estimate (NPMLE) of the distribution function from left-truncated and case 1 interval-censored data is inconsistent, in contrast to the consistency properties of the NPMLE from only left-truncated data or only interval-censored data. However, the conditional NPMLE is shown to be consistent. Numerical examples are provided to illustrate their finite sample properties.     

Estimating survival curves with left truncated and interval censored data via the ems algorithm

It is well-known that the nonparametric maximum likelihood estimator (NPMLE) of a survival function may severely underestimate the survival probabilities at very early times for left truncated data. This problem might be overcome by instead computing a smoothed nonparametric estimator (SNE) via the EMS algorithm. The close connection between the SNE and the maximum penalized likelihood estimator is also established. Extensive Monte Carlo simulations demonstrate the superior performance of the SNE over that of the NPMLE, in terms of either bias or variance, even for moderately large Samples. The methodology is illustrated with an application to the Massachusetts Health Care Panel Study dataset to estimate the probability of being functionally independent for non-poor male and female groups rcspectively.     

The random partition masking model for interval-censored and masked competing risks data

We study the nonparametric maximum likelihood estimate (NPMLE) of the cdf or sub-distribution functions of the failure time for the failure causes in a series system. The study is motivated by a cancer research data (from the Memorial Sloan-Kettering Cancer Center) with interval-censored time and masked failure cause. The NPMLE based on this data set suggests that the existing masking models are not appropriate. We propose a new model called the random partition masking model, which does not rely on the commonly used symmetry assumption (namely, given the failure cause, the probability of observing the masked failure causes is independent of the failure time; see Flehinger et al. Inference about defects in the presence of masking, Technometrics 38 (1996), pp. 247–255). The RPM model is easier to implement in simulation studies than the existing models. We discuss the algorithms for computing the NPMLE and study its asymptotic properties. Our simulation and data analysis indicate that the NPMLE is feasible for a moderate sample size.     

Consistency of the NPML Estimator in the Right-Censored Transformation Model

Abstract.  This paper studies the representation and large-sample consistency for non-parametric maximum likelihood estimators (NPMLEs) of an unknown baseline continuous cumulative-hazard-type function and parameter of group survival difference, based on right-censored two-sample survival data with marginal survival function assumed to follow a transformation model, a slight generalization of the class of frailty survival regression models. The paper's main theoretical results are existence and unique a.s. limit, characterized variationally, for large data samples of the NPMLE of baseline nuisance function in an appropriately defined neighbourhood of the true function when the group difference parameter is fixed, leading to consistency of the NPMLE when the difference parameter is fixed at a consistent estimator of its true value. The joint NPMLE is also shown to be consistent. An algorithm for computing it numerically, based directly on likelihood equations in place of the expectation-maximization (EM) algorithm, is illustrated with real data.     

A new algorithm for maximum likelihood estimation in normal scale-mixture generalized autoregressive conditional heteroskedastic models

In this paper, we propose a new generalized autoregressive conditional heteroskedastic (GARCH) model using infinite normal scale-mixtures which can suitably avoid order selection problems in the application of finite normal scale-mixtures. We discuss its theoretical properties and develop a two-stage algorithm for the maximum likelihood estimator to estimate the mixing distribution non-parametric maximum likelihood estimator (NPMLE) as well as GARCH parameters (two-stage MLE). For the estimation of a mixing distribution, we employ a fast computational algorithm proposed by Wang [On fast computation of the non-parametric maximum likelihood estimate of a mixing distribution. J R Stat Soc Ser B. 2007;69:185–198] under the gradient characterization of the non-parametric mixture likelihood. The GARCH parameters are then estimated either using the expectation-mazimization algorithm or general optimization scheme. In addition, we propose a new forecasting algorithm of value-at-risk (VaR) using the two-stage MLE and the NPMLE. Through a simulation study and real data analysis, we compare the performance of the two-stage MLE with the existing ones including quasi-maximum likelihood estimator based on the standard normal density and the finite normal mixture quasi maximum estimated-likelihood estimator (cf. Lee S, Lee T. Inference for Box–Cox transformed threshold GARCH models with nuisance parameters. Scand J Stat. 2012;39:568–589) in terms of the relative efficiency and accuracy of VaR forecasting.     

Estimating the survival function based on the semi-Markov model for dependent censoring

In this paper, we study a nonparametric maximum likelihood estimator (NPMLE) of the survival function based on a semi-Markov model under dependent censoring. We show that the NPMLE is asymptotically normal and achieves asymptotic nonparametric efficiency. We also provide a uniformly consistent estimator of the corresponding asymptotic covariance function based on an information operator. The finite-sample performance of the proposed NPMLE is examined with simulation studies, which show that the NPMLE has smaller mean squared error than the existing estimators and its corresponding pointwise confidence intervals have reasonable coverages. A real example is also presented.     

An inverse-probability-weighted approach to the estimation of distribution function with middle-censored data

In this article, we propose an inverse-probability-weighted (IPW) estimator of distribution function for middle-censored data. By Jammalamadaka and Mangalam (2003), the IPW estimator is the nonparametric maximum likelihood estimator (NPMLE) when all censored intervals contain at least one uncensored observation. The asymptotic properties of the IPW estimator are derived. A simulation study is conducted to compare the performance between the IPW estimator and the self-consistent estimator (SCE). Simulation results indicate that the performance of the IPW estimator is close to that of the SCE.     

Computing the log concave NPMLE for interval censored data

In analyzing interval censored data, a non-parametric estimator is often desired due to difficulties in assessing model fits. Because of this, the non-parametric maximum likelihood estimator (NPMLE) is often the default estimator. However, the estimates for values of interest of the survival function, such as the quantiles, have very large standard errors due to the jagged form of the estimator. By forcing the estimator to be constrained to the class of log concave functions, the estimator is ensured to have a smooth survival estimate which has much better operating characteristics than the unconstrained NPMLE, without needing to specify a parametric family or smoothing parameter. In this paper, we first prove that the likelihood can be maximized under a finite set of parameters under mild conditions, although the log likelihood function is not strictly concave. We then present an efficient algorithm for computing a local maximum of the likelihood function. Using our fast new algorithm, we present evidence from simulated current status data suggesting that the rate of convergence of the log-concave estimator is faster (between \(n^{2/5}\) and \(n^{1/2}\)) than the unconstrained NPMLE (between \(n^{1/3}\) and \(n^{1/2}\)).     

Constrained nonparametric maximum likelihood estimation of stochastically ordered survivor functions

This paper considers estimators of survivor functions subject to a stochastic ordering constraint based on right censored data. We present the constrained nonparametric maximum likelihood estimator (C‐NPMLE) of the survivor functions in one‐and two‐sample settings where the survivor distributions could be discrete or continuous and discuss the non‐uniqueness of the estimators. We also present a computationally efficient algorithm to obtain the C‐NPMLE. To address the possibility of non‐uniqueness of the C‐NPMLE of $S_1(t)$ when $S_1(t)\le S_2(t)$ , we consider the maximum C‐NPMLE (MC‐NPMLE) of $S_1(t)$ . In the one‐sample case with arbitrary upper bound survivor function $S_2(t)$ , we present a novel and efficient algorithm for finding the MC‐NPMLE of $S_1(t)$ . Dykstra ( 1982 ) also considered constrained nonparametric maximum likelihood estimation for such problems, however, as we show, Dykstra's method has an error and does not always give the C‐NPMLE. We corrected this error and simulation shows improvement in efficiency compared to Dykstra's estimator. Confidence intervals based on bootstrap methods are proposed and consistency of the estimators is proved. Data from a study on larynx cancer are analysed to illustrate the method. The Canadian Journal of Statistics 40: 22–39; 2012 © 2012 Statistical Society of Canada     

An Adjustment to Improve the Bivariate Survivor Function Repaired NPMLE

We recently proposed a representation of the bivariate survivor function as a mapping of the hazard function for truncated failure time variates. The representation led to a class of estimators that includes van der Laan’s repaired nonparametric maximum likelihood estimator (NPMLE) as an important special case. We proposed a Greenwood-like variance estimator for the repaired NPMLE but found somewhat poor agreement between the empirical variance estimates and these analytic estimates for the sample sizes and bandwidths considered in our simulation study. The simulation results also confirmed those of others in showing slightly inferior performance for the repaired NPMLE compared to other competing estimators as well as a sensitivity to bandwidth choice in moderate sized samples. Despite its attractive asymptotic properties, the repaired NPMLE has drawbacks that hinder its practical application. This paper presents a modification of the repaired NPMLE that improves its performance in moderate sized samples and renders it less sensitive to the choice of bandwidth. Along with this modified estimator, more extensive simulation studies of the repaired NPMLE and Greenwood-like variance estimates are presented. The methods are then applied to a real data example. This revised version was published online in September 2005 with a correction to the second author's name.     

A Nonparametric Estimator of Survival Functions for Arbitrarily Truncated and Censored Data

It is well-known that the nonparametric maximum likelihood estimator (NPMLE) may severely under-estimate the survival function with left truncated data. Based on the Nelson estimator (for right censored data) and self-consistency we suggest a nonparametric estimator of the survival function, the iterative Nelson estimator (INE), for arbitrarily truncated and censored data, where only few nonparametric estimators are available. By simulation we show that the INE does well in overcoming the under-estimation of the survival function from the NPMLE for left-truncated and interval-censored data. An interesting application of the INE is as a diagnostic tool for other estimators, such as the monotone MLE or parametric MLEs. The methodology is illustrated by application to two real world problems: the Channing House and the Massachusetts Health Care Panel Study data sets.     

Nonparametric Estimation of the Bivariate Distribution Function with Doubly Truncated Data

Doubly truncated data play an important role in the statistical analysis of astronomical observations as well as in survival analysis. In this article, using inverse-probability-weighted (IPW) approaches, we derive the nonparametric maximum likelihood estimator (NPMLE) of joint distribution function with bivariate doubly truncated data. The asymptotic properties of the NPMLE are established. A simulation study is conducted to investigate the performance of the NPMLE.     

Goodness-of-fit tests for one-shot device accelerated life testing data

In this article, we develop a formal goodness-of-fit testing procedure for one-shot device testing data, in which each observation in the sample is either left censored or right censored. Such data are also called current status data. We provide an algorithm for calculating the nonparametric maximum likelihood estimate (NPMLE) of the unknown lifetime distribution based on such data. Then, we consider four different test statistics that can be used for testing the goodness-of-fit of accelerated failure time (AFT) model by the use of samples of residuals: a chi-square-type statistic based on the difference between the empirical and expected numbers of failures at each inspection time; two other statistics based on the difference between the NPMLE of the lifetime distribution obtained from one-shot device testing data and the distribution specified under the null hypothesis; as a final statistic, we use White's idea of comparing two estimators of the Fisher Information (FI) to propose a test statistic. We then compare these tests in terms of power, and draw some conclusions. Finally, we present an example to illustrate the proposed tests.     

Nonparametric estimation of the cumulative intensities in an interval censored competing risks model

The nonparametric maximum likelihood estimation (NPMLE) of the distribution function from the interval censored (IC) data has been extensively studied in the extant literature. The NPMLE was also developed for the subdistribution functions in an IC competing risks model and in an illness-death model under various interval-censoring scenarios. But the important problem of estimation of the cumulative intensities (CIs) in the interval-censored models has not been considered previously. We develop the NPMLE of the CI in a simple alive/dead model and of the CIs in a competing risks model. Assuming that data are generated by a discrete and finite mixed case interval censoring mechanism we provide a discussion and the simulation study of the asymptotic properties of the NPMLEs of the CIs. In particular we show that they are asymptotically unbiased; in contrast the ad hoc estimators presented in extant literature are substantially biased. We illustrate our methods with the data from a prospective cohort study on the longevity of dental veneers.     

Nonparametric estimators of survival function under the mixed case interval-censored model with left truncation

It is well known that the nonparametric maximum likelihood estimator (NPMLE) can severely underestimate the survival probabilities at early times for left-truncated and interval-censored (LT-IC) data. For arbitrarily truncated and censored data, Pan and Chappel (JAMA Stat Probab Lett 38:49–57, 1998a, Biometrics 54:1053–1060, 1998b) proposed a nonparametric estimator of the survival function, called the iterative Nelson estimator (INE). Their simulation study showed that the INE performed well in overcoming the under-estimation of the survival function from the NPMLE for LT-IC data. In this article, we revisit the problem of inconsistency of the NPMLE. We point out that the inconsistency is caused by the likelihood function of the left-censored observations, where the left-truncated variables are used as the left endpoints of censoring intervals. This can lead to severe underestimation of the survival function if the NPMLE is obtained using Turnbull’s (JAMA 38:290–295, 1976) EM algorithm. To overcome this problem, we propose a modified maximum likelihood estimator (MMLE) based on a modified likelihood function, where the left endpoints of censoring intervals for left-censored observations are the maximum of left-truncated variables and the estimated left endpoint of the support of the left-censored times. Simulation studies show that the MMLE performs well for finite sample and outperforms both the INE and NPMLE.

     

Nonparametric estimation of the bivariate cdf for arbitrarily censored data

Right, left or interval censored multivariate data can be represented by an intersection graph. Focussing on the bivariate case, the authors relate the structure of such an intersection graph to the support of the nonparametric maximum likelihood estimate (NPMLE) of the cumulative distribution function (CDF) for such data. They distinguish two types of non‐uniqueness of the NPMLE: representational, arising when the likelihood is unaffected by the distribution of the estimated probability mass within regions, and mixture, arising when the masses themselves are not unique. The authors provide a brief overview of estimation techniques and examine three data sets.