Multivariate outlier detection in incomplete survey data: the epidemic algorithm and transformed rank correlations

Summary.  As a part of the EUREDIT project new methods to detect multivariate outliers in incomplete survey data have been developed. These methods are the first to work with sampling weights and to be able to cope with missing values. Two of these methods are presented here. The epidemic algorithm simulates the propagation of a disease through a population and uses extreme infection times to find outlying observations. Transformed rank correlations are robust estimates of the centre and the scatter of the data. They use a geometric transformation that is based on the rank correlation matrix. The estimates are used to define a Mahalanobis distance that reveals outliers. The two methods are applied to a small data set and to one of the evaluation data sets of the EUREDIT project.     

Estimating household structure in ancient China by using historical data: a latent class analysis of partially missing patterns

Summary.  Social data often contain missing information. The problem is inevitably severe when analysing historical data. Conventionally, researchers analyse complete records only. Listwise deletion not only reduces the effective sample size but also may result in biased estimation, depending on the missingness mechanism. We analyse household types by using population registers from ancient China (618–907 AD) by comparing a simple classification, a latent class model of the complete data and a latent class model of the complete and partially missing data assuming four types of ignorable and non-ignorable missingness mechanisms. The findings show that either a frequency classification or a latent class analysis using the complete records only yielded biased estimates and incorrect conclusions in the presence of partially missing data of a non-ignorable mechanism. Although simply assuming ignorable or non-ignorable missing data produced consistently similarly higher estimates of the proportion of complex households, a specification of the relationship between the latent variable and the degree of missingness by a row effect uniform association model helped to capture the missingness mechanism better and improved the model fit.     

Multiple imputation of missing data with ante-dependence covariance structure

A controlled clinical trial was conducted to investigate the efficacy effect of a chemical compound in the treatment of Premenstrual Dysphoric Disorder (PMDD). The data from the trial showed a non-monotone pattern of missing data and an ante-dependence covariance structure. A new analytical method for imputing the missing data with the ante-dependence covariance is proposed. The PMDD data are analysed by the non-imputation method and two imputation methods: the proposed method and the MCMC method.     

Integrated methodology for multiple systems estimation and record linkage using a missing data formulation

There are now three essentially separate literatures on the topics of multiple systems estimation, record linkage, and missing data. But in practice the three are intimately intertwined. For example, record linkage involving multiple data sources for human populations is often carried out with the expressed goal of developing a merged database for multiple system estimation (MSE). Similarly, one way to view both the record linkage and MSE problems is as ones involving the estimation of missing data. This presentation highlights the technical nature of these interrelationships and provides a preliminary effort at their integration.     

Robust nonparametric estimation with missing data

In this paper, under a nonparametric regression model, we introduce two families of robust procedures to estimate the regression function when missing data occur in the response. The first proposal is based on a local M$M$-functional applied to the conditional distribution function estimate adapted to the presence of missing data. The second proposal imputes the missing responses using the local M$M$-smoother based on the observed sample and then estimates the regression function with the completed sample. We show that the robust procedures considered are consistent and asymptotically normally distributed. A robust procedure to select the smoothing parameter is also discussed.     

Permuting incomplete paired data: a novel exact and asymptotic correct randomization test

Various statistical tests have been developed for testing the equality of means in matched pairs with missing values. However, most existing methods are commonly based on certain distributional assumptions such as normality, 0-symmetry or homoscedasticity of the data. The aim of this paper is to develop a statistical test that is robust against deviations from such assumptions and also leads to valid inference in case of heteroscedasticity or skewed distributions. This is achieved by applying a clever randomization approach to handle missing data. The resulting test procedure is not only shown to be asymptotically correct but is also finitely exact if the distribution of the data is invariant with respect to the considered randomization group. Its small sample performance is further studied in an extensive simulation study and compared to existing methods. Finally, an illustrative data example is analysed.     

Convergence rate of wavelet density estimator with data missing randomly when covariables are present

In this article, we study global L₂ error of non linear wavelet estimator of density in the Besov space B^s_pq for missing data model when covariables are present and prove that the estimator can achieve the optimal rate of convergence, which is similar to the result studied by Donoho et al. (1996) Donoho, D.L., Johnstone, I.M., Kerkyacharian, G., Picard, D. (1996). Density estimation by wavelet thresholding. Ann. Stat. 24:508–539.[Crossref], [Web of Science ®] , [Google Scholar] in complete independent data case with term-by-term thresholding of the empirical wavelet coefficients. Finite-sample behavior of the proposed estimator is explored via simulations.     

Ignorability conditions for frequentist non parametric analysis of conditional distributions with incomplete data

Rubin (1976 Rubin, D.B. (1976). Inference and missing data. Biometrika 63(3):581–592.[Crossref], [Web of Science ®] , [Google Scholar]) derived general conditions under which inferences that ignore missing data are valid. These conditions are sufficient but not generally necessary, and therefore may be relaxed in some special cases. We consider here the case of frequentist estimation of a conditional cdf subject to missing outcomes. We partition a set of data into outcome, conditioning, and latent variables, all of which potentially affect the probability of a missing response. We describe sufficient conditions under which a complete-case estimate of the conditional cdf of the outcome given the conditioning variable is unbiased. We use simulations on a renal transplant data set (Dienemann et al.) to illustrate the implications of these results.     

Generalized empirical likelihood inference in partially linear model for longitudinal data with missing response variables and error-prone covariates

In this article, we consider statistical inference for longitudinal partial linear models when the response variable is sometimes missing with missingness probability depending on the covariate that is measured with error. A generalized empirical likelihood (GEL) method is proposed by combining correction attenuation and quadratic inference functions. The method that takes into consideration the correlation within groups is used to estimate the regression coefficients. Furthermore, residual-adjusted empirical likelihood (EL) is employed for estimating the baseline function so that undersmoothing is avoided. The empirical log-likelihood ratios are proven to be asymptotically Chi-squared, and the corresponding confidence regions for the parameters of interest are then constructed. Compared with methods based on NAs, the GEL does not require consistent estimators for the asymptotic variance and bias. The numerical study is conducted to compare the performance of the EL and the normal approximation-based method, and a real example is analysed.