Random weighting-based quantile estimation via importance resampling

Abstract

This paper presents a new method to estimate the quantiles of generic statistics by combining the concept of random weighting with importance resampling. This method converts the problem of quantile estimation to a dual problem of tail probabilities estimation. Random weighting theories are established to calculate the optimal resampling weights for estimation of tail probabilities via sequential variance minimization. Subsequently, the quantile estimation is constructed by using the obtained optimal resampling weights. Experimental results on real and simulated data sets demonstrate that the proposed random weighting method can effectively estimate the quantiles of generic statistics.     

Confidence and prediction intervals based on interpolated records

In several statistical problems, nonparametric confidence intervals for population quantiles can be constructed and their coverage probabilities can be computed exactly, but cannot in general be rendered equal to a pre-determined level. The same difficulty arises for coverage probabilities of nonparametric prediction intervals for future observations. One solution to this difficulty is to interpolate between intervals which have the closest coverage probability from above and below to the pre-determined level. In this paper, confidence intervals for population quantiles are constructed based on interpolated upper and lower records. Subsequently, prediction intervals are obtained for future upper records based on interpolated upper records. Additionally, we derive upper bounds for the coverage error of these confidence and prediction intervals. Finally, our results are applied to some real data sets. Also, a comparison via a simulation study is done with similar classical intervals obtained before.     

On estimation of conditional density models with two-phase sampling

Suppose that the conditional density of a response variable given a vector of explanatory variables is parametrically modelled, and that data are collected by a two-phase sampling design. First, a simple random sample is drawn from the population. The stratum membership in a finite number of strata of the response and explanatory variables is recorded for each unit. Second, a subsample is drawn from the phase-one sample such that the selection probability is determined by the stratum membership. The response and explanatory variables are fully measured at this phase. We synthesize existing results on nonparametric likelihood estimation and present a streamlined approach for the computation and the large sample theory of profile likelihood in four different situations. The amount of information in terms of data and assumptions varies depending on whether the phase-one data are retained, the selection probabilities are known, and/or the stratum probabilities are known. We establish and illustrate numerically the order of efficiency among the maximum likelihood estimators, according to the amount of information utilized, in the four situations.     

Noninformative nonparametric quantile estimation for simple random samples

For noninformative nonparametric estimation of finite population quantiles under simple random sampling, estimation based on the Polya posterior is similar to estimation based on the Bayesian approach developed by Ericson (J. Roy. Statist. Soc. Ser. B 31 (1969) 195) in that the Polya posterior distribution is the limit of Ericson's posterior distributions as the weight placed on the prior distribution diminishes. Furthermore, Polya posterior quantile estimates can be shown to be admissible under certain conditions. We demonstrate the admissibility of the sample median as an estimate of the population median under such a set of conditions. As with Ericson's Bayesian approach, Polya posterior-based interval estimates for population quantiles are asymptotically equivalent to the interval estimates obtained from standard frequentist approaches. In addition, for small to moderate sized populations, Polya posterior-based interval estimates for quantiles of a continuous characteristic of interest tend to agree with the standard frequentist interval estimates.     

Bias in transition-specific survival and movement probabilities estimated using capture-recapture data

Transition probabilities can be estimated when capture-recapture data are available from each stratum on every capture occasion using a conditional likelihood approach with the Arnason-Schwarz model. To decompose the fundamental transition probabilities into derived parameters, all movement probabilities must sum to 1 and all individuals in stratum r at time i must have the same probability of survival regardless of which stratum the individual is in at time i + 1. If movement occurs among strata at the end of a sampling interval, survival rates of individuals from the same stratum are likely to be equal. However, if movement occurs between sampling periods and survival rates of individuals from the same stratum are not the same, estimates of stratum survival can be confounded with estimates of movement causing both estimates to be biased. Monte Carlo simulations were made of a three-sample model for a population with two strata using SURVIV. When differences were created in transition-specific survival rates for survival rates from the same stratum, relative bias was <2% in estimates of stratum survival and capture rates but relative bias in movement rates was much higher and varied. The magnitude of the relative bias in the movement estimate depended on the relative difference between the transition-specific survival rates and the corresponding stratum survival rate. The direction of the bias in movement rate estimates was opposite to the direction of this difference. Increases in relative bias due to increasing heterogeneity in probabilities of survival, movement and capture were small except when survival and capture probabilities were positively correlated within individuals.     

Nonparametric estimation of extreme conditional quantiles

The estimation of extreme conditional quantiles is an important issue in different scientific disciplines. Up to now, the extreme value literature focused mainly on estimation procedures based on independent and identically distributed samples. Our contribution is a two-step procedure for estimating extreme conditional quantiles. In a first step nonextreme conditional quantiles are estimated nonparametrically using a local version of [Koenker, R. and Bassett, G. (1978). Regression quantiles. Econometrica, 46, 33–50.] regression quantile methodology. Next, these nonparametric quantile estimates are used as analogues of univariate order statistics in procedures for extreme quantile estimation. The performance of the method is evaluated for both heavy tailed distributions and distributions with a finite right endpoint using a small sample simulation study. A bootstrap procedure is developed to guide in the selection of an optimal local bandwidth. Finally the procedure is illustrated in two case studies.     

Simultaneous Pitman closeness of progressively type-II right-censored order statistics to population quantiles

In this work, we extend prior results concerning the simultaneous Pitman closeness of order statistics (OS) to population quantiles. By considering progressively type-II right-censored samples, we derive expressions for the simultaneous closeness probabilities of the progressively censored OS to population quantiles. Explicit expressions are deduced for the cases when the underlying distribution has bounded and unbounded supports. Illustrations are provided for the cases of exponential, uniform and normal distributions for various progressive type-II right-censoring schemes and different quantiles. Finally, an extension to the case of generalized OS is outlined.     

Estimating and comparing adverse event probabilities in the presence of varying follow-up times and competing events

Safety analyses of adverse events (AEs) are important in assessing benefit–risk of therapies but are often rather simplistic compared to efficacy analyses. AE probabilities are typically estimated by incidence proportions, sometimes incidence densities or Kaplan–Meier estimation are proposed. These analyses either do not account for censoring, rely on a too restrictive parametric model, or ignore competing events. With the non-parametric Aalen-Johansen estimator as the “gold standard”, that is, reference estimator, potential sources of bias are investigated in an example from oncology and in simulations, for both one-sample and two-sample scenarios. The Aalen-Johansen estimator serves as a reference, because it is the proper non-parametric generalization of the Kaplan–Meier estimator to multiple outcomes. Because of potential large variances at the end of follow-up, comparisons also consider further quantiles of the observed times. To date, consequences for safety comparisons have hardly been investigated, the impact of using different estimators for group comparisons being unclear. For example, the ratio of two both underestimating or overestimating estimators may not be comparable to the ratio of the reference, and our investigation also considers the ratio of AE probabilities. We find that ignoring competing events is more of a problem than falsely assuming constant hazards by the use of the incidence density and that the choice of the AE probability estimator is crucial for group comparisons.     

On Pitman's measure of closeness of k-records

Pitman closeness of both the upper and lower k-record statistics to the population quantiles of a location–scale family of distributions is studied. For the population median, the Pitman-closest k-record is also determined. In the case of symmetric distributions, the Pitman closeness probabilities of k-record statistics are shown to be distribution-free, and explicit expressions are also derived for these probabilities. Exact expressions are derived for the required probabilities for uniform and exponential distributions. Numerical results are given for these families and also the Pitman-closest k-record is determined.     

Empirical likelihood intervals for the population mean and quantiles based on balanced ranked set samples

We consider nonparametric interval estimation for the population mean and quantiles based on a ranked set sample. The asymptotic distributions of the empirical log likelihood ratio statistic for the mean and quantiles are derived. Interval estimates of the population mean and quantiles are obtained by inverting the likelihood ratio statistic. Simulations are carried out to investigate and compare the performance of the empirical likelihood intervals with other known intervals.     

ESTIMATING WITHIN-HOUSEHOLD CONTACT NETWORKS FROM EGOCENTRIC DATA

Acute respiratory diseases are transmitted over networks of social contacts. Large-scale simulation models are used to predict epidemic dynamics and evaluate the impact of various interventions, but the contact behavior in these models is based on simplistic and strong assumptions which are not informed by survey data. These assumptions are also used for estimating transmission measures such as the basic reproductive number and secondary attack rates. Development of methodology to infer contact networks from survey data could improve these models and estimation methods. We contribute to this area by developing a model of within-household social contacts and using it to analyze the Belgian POLYMOD data set, which contains detailed diaries of social contacts in a 24-hour period. We model dependency in contact behavior through a latent variable indicating which household members are at home. We estimate age-specific probabilities of being at home and age-specific probabilities of contact conditional on two members being at home. Our results differ from the standard random mixing assumption. In addition, we find that the probability that all members contact each other on a given day is fairly low: 0.49 for households with two 0-5 year olds and two 19-35 year olds, and 0.36 for households with two 12-18 year olds and two 36+ year olds. We find higher contact rates in households with 2-3 members, helping explain the higher influenza secondary attack rates found in households of this size.     

Simultaneous Tolerance Limits and a Unified Approach to some Nonparametric Theory

In this paper nonparametric simultaneous tolerance limits are developed using rectangle probabilities for uniform order statistics. Consideration is given to the handling of censored data, and some comparisons are made with the parametric normal theory. The nonparametric regional estimation techniques of (i) confidence bands for a distribution function, (ii) simultaneous confidence intervals for quantiles and (iii) simultaneous tolerance limits are unified. A Bayesian approach is also discussed.     

Semiparametric estimation of regression quantiles with application to standardizing weight for height and age in US children

The appropriate interpretation of measurements often requires standardization for concomitant factors. For example, standardization of weight for both height and age is important in obesity research and in failure-to-thrive research in children. Regression quantiles from a reference population afford one intuitive and popular approach to standardization. Current methods for the estimation of regression quantiles can be classified as nonparametric with respect to distributional assumptions or as fully parametric. We propose a semiparametric method where we model the mean and variance as flexible regression spline functions and allow the unspecified distribution to vary smoothly as a function of covariates. Similarly to Cole and Green, our approach provides separate estimates and summaries for location, scale and distribution. However, similarly to Koenker and Bassett, we do not assume any parametric form for the distribution. Estimation for either cross-sectional or longitudinal samples is obtained by using estimating equations for the location and scale functions and through local kernel smoothing of the empirical distribution function for standardized residuals. Using this technique with data on weight, height and age for females under 3 years of age, we find that there is a close relationship between quantiles of weight for height and age and quantiles of body mass index (BMI=weight/height²) for age in this cohort.     

A MODEL OF SYSTEMATIC SAMPLING WITH UNEQUAL PROBABILITIES

A new method is described of drawing, without replacement, two sample units per stratum from any population. The method is developed from a consideration of the asymptotic properties of systematic sampling with unequal probabilities, as the sizes of the population units tend to zero. The essential properties of this method are very easily analysed. They also converge, over a large number of strata, to those of systematic sampling from the same strata with their population units arranged in random order. In proving this, the assumption is made that the underlying population is of the type to which it is appropriate to apply ratio estimation. The sampling method described is, however, simple enough to commend itself as an alternative to systematic sampling when the underlying population is not of this type. Consideration is given to the case where the sizes of some of the population units exceed the skip interval.     

Empirical Likelihood Inference for Population Quantiles with Unbalanced Ranked Set Samples

We consider nonparametric interval estimation for the population quantiles based on unbalanced ranked set samples. We derived the large sample distribution of the empirical log likelihood ratio statistic for the quantiles. Approximate intervals for quantiles are obtained by inverting the likelihood ratio statistic. The performance of the empirical likelihood interval is investigated and compared with the performance of the intervals based on the ranked set sample order statistics.     

A two-phase sampling scheme and πps designs

A two-phase approach for sampling with unequal inclusions probabilities and fixed sample size is presented. The expansion estimator using target inclusion probabilities is suggested for estimation of population parameters. As an alternative, the estimator for two-phase sampling can be used for estimation. Inclusion probabilities are shown to be asymptotically equivalent to the targeted inclusion probabilities. By means of simulation associated estimators are shown to work well with respect to bias and precision.     

Pitman closeness of record values from two sequences to population quantiles

Suppose upper records from two independent sequences from iid continuous random variables from the same distribution are observed. Pitman's measure of closeness of these statistics to population quantiles of the parent distribution is studied and various exact expressions are derived. For symmetric distributions, Pitman closeness probabilities of records to median are also obtained. Examples including exponential and uniform distributions are discussed. Numerical evaluations are presented to illustrate all the results developed here.     

Simultaneous Closeness of k-Records

In this article, we study the simultaneous Pitman closeness of upper (and lower) k-records to a common parameter of the parent distribution and obtain general expressions for the corresponding probabilities. Since the usual record values are contained in the k-records, the corresponding results for the usual records can be deduced as special cases. The results are then applied to location-scale families with an emphasis on population quantiles. Exact expressions are derived for some common distributions such as exponential and uniform. In each case, the simultaneous closest k-record to a population quantile is determined.     

Generalised regression estimation via the bootstrap

A generalised regression estimation procedure is proposed that can lead to much improved estimation of population characteristics, such as quantiles, variances and coefficients of variation. The method involves conditioning on the discrepancy between an estimate of an auxiliary parameter and its known population value. The key distributional assumption is joint asymptotic normality of the estimates of the target and auxiliary parameters. This assumption implies that the relationship between the estimated target and the estimated auxiliary parameters is approximately linear with coefficients determined by their asymptotic covariance matrix. The main contribution of this paper is the use of the bootstrap to estimate these coefficients, which avoids the need for parametric distributional assumptions. First‐order correct conditional confidence intervals based on asymptotic normality can be improved upon using quantiles of a conditional double bootstrap approximation to the distribution of the studentised target parameter estimate.     

Best Invariant and Minimax Estimation of Quantiles in Finite Populations

The theoretical literature on quantile and distribution function estimation in infinite populations is very rich, and invariance plays an important role in these studies. This is not the case for the commonly occurring problem of estimation of quantiles in finite populations. The latter is more complicated and interesting because an optimal strategy consists not only of an estimator, but also of a sampling design, and the estimator may depend on the design and on the labels of sampled individuals, whereas in iid sampling, design issues and labels do not exist.We study the estimation of finite population quantiles, with emphasis on estimators that are invariant under the group of monotone transformations of the data, and suitable invariant loss functions. Invariance under the finite group of permutation of the sample is also considered. We discuss nonrandomized and randomized estimators, best invariant and minimax estimators, and sampling strategies relative to different classes. Invariant loss functions and estimators in finite population sampling have a nonparametric flavor, and various natural combinatorial questions and tools arise as a result.