Data skeletons: simultaneous estimation of multiple quantiles for massive streaming datasets with applications to density estimation

We consider the problem of density estimation when the data is in the form of a continuous stream with no fixed length. In this setting, implementations of the usual methods of density estimation such as kernel density estimation are problematic. We propose a method of density estimation for massive datasets that is based upon taking the derivative of a smooth curve that has been fit through a set of quantile estimates. To achieve this, a low-storage, single-pass, sequential method is proposed for simultaneous estimation of multiple quantiles for massive datasets that form the basis of this method of density estimation. For comparison, we also consider a sequential kernel density estimator. The proposed methods are shown through simulation study to perform well and to have several distinct advantages over existing methods.     

Minimax estimation of a probability of success under LINEX loss

Minimax estimation of a binomial probability under LINEX loss function is considered. It is shown that no equalizer estimator is available in the statistical decision problem under consideration. It is pointed out that the problem can be solved by determining the Bayes estimator with respect to a least favorable distribution having finite support. In this situation, the optimal estimator and the least favorable distribution can be determined only by using numerical methods. Some properties of the minimax estimators and the corresponding least favorable prior distributions are provided depending on the parameters of the loss function. The properties presented are exploited in computing the minimax estimators and the least favorable distributions. The results obtained can be applied to determine minimax estimators of a cumulative distribution function and minimax estimators of a survival function.     

Non-parametric estimation of population size from capture–recapture data when the capture probability depends on a covariate

Summary.  In capture–recapture experiments the capture probabilities may depend on individual covariates such as an individual's weight or age. Typically this dependence is modelled through simple parametric functions of the covariates. Here we first demonstrate that misspecification of the model can produce biased estimates and subsequently develop a non-parametric procedure to estimate the functional relationship between the probability of capture and a single covariate. This estimator is then incorporated in a Horvitz–Thompson estimator to estimate the size of the population. The resulting estimators are evaluated in a simulation study and applied to a data set on captures of the Mountain Pygmy Possum.     

Comparison of different estimation techniques for portfolio selection

The main problem in applying the mean-variance portfolio selection consists of the fact that the first two moments of the asset returns are unknown. In practice the optimal portfolio weights have to be estimated. This is usually done by replacing the moments by the classical unbiased sample estimators. We provide a comparison of the exact and the asymptotic distributions of the estimated portfolio weights as well as a sensitivity analysis to shifts in the moments of the asset returns. Furthermore we consider several types of shrinkage estimators for the moments. The corresponding estimators of the portfolio weights are compared with each other and with the portfolio weights based on the sample estimators of the moments. We show how the uncertainty about the portfolio weights can be introduced into the performance measurement of trading strategies. The methodology explains the bad out-of-sample performance of the classical Markowitz procedures.     

Search for optimal designs in a three stagenested random model

In this paper we define a class of unbalanced designs, denoted by C_k,s,t, for estimating the components of variance in a k-stage nested random effects linear model. This class contains many of the designs proposed in the literature for nested components of variance models. We focus on the three-state model and discuss the determination of locally optimal designs within this class using a systematic computer search. For large sample sizes we show that approximate optimal designs may be obtained using a limit argument combined with numerical optimization. A comparison of our designs with previously published designs suggests that, in many cases, our designs result in substantial gains in efficiency.     

Influence of human immunodeficiency virus infection on neurological impairment: an analysis of longitudinal binary data with informative drop-out

A study to investigate the human immunodeficiency virus (HIV) status on the course of neurological impairment, conducted by the HIV Center at Columbia University, followed a cohort of HIV positive and negative gay men for 5 years and assessed the presence or absence of neurological impairment every 6 months. Almost half of the subjects dropped out before the end of the study for reasons that might have been related to the missing neurological data. We propose likelihood-based methods for analysing such binary longitudinal data under informative and non-informative drop-out. A transition model is assumed for the binary response, and several models for the drop-out processes are considered which are functions of the response variable (neurological impairment). The likelihood ratio test is used to compare models with informative and non-informative drop-out mechanisms. Using simulations, we investigate the percentage bias and mean-squared error (MSE) of the parameter estimates in the transition model under various assumptions for the drop-out. We find evidence for informative drop-out in the study, and we illustrate that the bias and MSE for the parameters of the transition model are not directly related to the observed drop-out or missing data rates. The effect of HIV status on the neurological impairment is found to be statistically significant under each of the models considered for the drop-out, although the regression coefficient may be biased in certain cases. The presence and relative magnitude of the bias depend on factors such as the probability of drop-out conditional on the presence of neurological impairment and the prevalence of neurological impairment in the population under study.     

Modelling Variability in the Branching Structure of Strawberry Inflorescences

The branching structure of inflorescences of the cultivated strawberry ( Fragaria × ananassa Duch.) is very variable. This paper demonstrates that some aspects of this variability are well described by a simple stochastic model of branching that has two adjustable parameters. The model is shown to provide a good fit to data from a set of almost 700 inflorescences of the cultivar Elsanta, collected over two successive years. For one parameter the maximum likelihood estimator is a moment estimator which is fully efficient even if the detailed branching structure of the inflorescences is not recorded. This parameter provides a convenient summary of branching vigour. The maximum likelihood estimator of the second parameter must be determined iteratively and can be quite inefficient unless the full branching structure is recorded. The model demonstrates that branching structure is affected by the order in which inflorescences emerge on the plant.     

Venezuelan Rainfall Data Analysed by Using a Bayesian Space–time Model

We consider a set of data from 80 stations in the Venezuelan state of Guárico consisting of accumulated monthly rainfall in a time span of 16 years. The problem of modelling rainfall accumulated over fixed periods of time and recorded at meteorological stations at different sites is studied by using a model based on the assumption that the data follow a truncated and transformed multivariate normal distribution. The spatial correlation is modelled by using an exponentially decreasing correlation function and an interpolating surface for the means. Missing data and dry periods are handled within a Markov chain Monte Carlo framework using latent variables. We estimate the amount of rainfall as well as the probability of a dry period by using the predictive density of the data. We considered a model based on a full second-degree polynomial over the spatial co-ordinates as well as the first two Fourier harmonics to describe the variability during the year. Predictive inferences on the data show very realistic results, capturing the typical rainfall variability in time and space for that region. Important extensions of the model are also discussed.     

Shrinkage and Orthogonal Decomposition

ABSTRACT. The problem of estimating the mean of a multivariate normal distribution when the parameter space allows an orthogonal decomposition is discussed. Risk functions and lower bounds for a class of shrinkage estimators that includes Stein's estimator are derived, and an improvement on Stein's estimator that takes advantage of the orthogonal decomposition is introduced. Uniform asymptotics related to Pinsker's minimax risk is derived and we give conditions for attaining the lower risk bound. Special cases including regression and analysis of variance are discussed.