VARIANCE ESTIMATION IN TWO-PHASE SAMPLING

Two‐phase sampling is often used for estimating a population total or mean when the cost per unit of collecting auxiliary variables, x, is much smaller than the cost per unit of measuring a characteristic of interest, y. In the first phase, a large sample s₁ is drawn according to a specific sampling design p(s₁) , and auxiliary data x are observed for the units i∈s₁ . Given the first‐phase sample s₁ , a second‐phase sample s₂ is selected from s₁ according to a specified sampling design {p(s₂∣s₁) } , and (y, x) is observed for the units i∈s₂ . In some cases, the population totals of some components of x may also be known. Two‐phase sampling is used for stratification at the second phase or both phases and for regression estimation. Horvitz–Thompson‐type variance estimators are used for variance estimation. However, the Horvitz–Thompson ( Horvitz & Thompson, J. Amer. Statist. Assoc. 1952 ) variance estimator in uni‐phase sampling is known to be highly unstable and may take negative values when the units are selected with unequal probabilities. On the other hand, the Sen–Yates–Grundy variance estimator is relatively stable and non‐negative for several unequal probability sampling designs with fixed sample sizes. In this paper, we extend the Sen–Yates–Grundy ( Sen , J. Ind. Soc. Agric. Statist. 1953; Yates & Grundy , J. Roy. Statist. Soc. Ser. B 1953) variance estimator to two‐phase sampling, assuming fixed first‐phase sample size and fixed second‐phase sample size given the first‐phase sample. We apply the new variance estimators to two‐phase sampling designs with stratification at the second phase or both phases. We also develop Sen–Yates–Grundy‐type variance estimators of the two‐phase regression estimators that make use of the first‐phase auxiliary data and known population totals of some of the auxiliary variables.     

A Class of Sampling Two Units with Probability Proportional to Size

A class of sampling two units without replacement with inclusion probability proportional to size is proposed in this article. Many different well known probability proportional to size sampling designs are special cases from this class. The first and second inclusion probabilities of this class satisfy important properties and provide a non-negative variance estimator of the Horvitz and Thompson estimator for the population total. Suitable choice for the first and second inclusion probabilities from this class can be used to reduce the variance estimator of the Horvitz and Thompson estimator. Comparisons between different proportional to size sampling designs through real data and artificial examples are given. Examples show that the minimum variance of the Horvitz and Thompson estimator obtained from the proposed design is not attainable for the most cases at any of the well known designs.     

Variance Estimation and Asymptotic Confidence Bands for the Mean Estimator of Sampled Functional Data with High Entropy Unequal Probability Sampling Designs

For fixed size sampling designs with high entropy, it is well known that the variance of the Horvitz–Thompson estimator can be approximated by the Hájek formula. The interest of this asymptotic variance approximation is that it only involves the first order inclusion probabilities of the statistical units. We extend this variance formula when the variable under study is functional, and we prove, under general conditions on the regularity of the individual trajectories and the sampling design, that we can get a uniformly convergent estimator of the variance function of the Horvitz–Thompson estimator of the mean function. Rates of convergence to the true variance function are given for the rejective sampling. We deduce, under conditions on the entropy of the sampling design, that it is possible to build confidence bands whose coverage is asymptotically the desired one via simulation of Gaussian processes with variance function given by the Hájek formula. Finally, the accuracy of the proposed variance estimator is evaluated on samples of electricity consumption data measured every half an hour over a period of 1 week.     

An Optimal Procedure for Sample Coordination Using Multiple Objective Functions and Nearest Proportional to IPPS Size Sampling Designs

The sample coordination problem involves maximization or minimization of overlap of sampling units in different/repeated surveys. Several optimal techniques using transportation theory, controlled rounding, and controlled selection have been suggested in literature to solve the sample coordination problem. In this article, using the multiple objective programming, we propose a method for sample coordination which facilitates variance estimation using the Horvitz–Thompson estimator. The proposed procedure can be applied to any two-sample surveys having identical universe and stratification. Some examples are discussed to demonstrate the utility of the proposed procedure.     

Spatially Balanced Sampling of Continuous Populations

When sampling from a continuous population (or distribution), we often want a rather small sample due to some cost attached to processing the sample or to collecting information in the field. Moreover, a probability sample that allows for design‐based statistical inference is often desired. Given these requirements, we want to reduce the sampling variance of the Horvitz–Thompson estimator as much as possible. To achieve this, we introduce different approaches to using the local pivotal method for selecting well‐spread samples from multidimensional continuous populations. The results of a simulation study clearly indicate that we succeed in selecting spatially balanced samples and improve the efficiency of the Horvitz–Thompson estimator.     

Estimating prediction error for complex samples

With a growing interest in using non-representative samples to train prediction models for numerous outcomes it is necessary to account for the sampling design that gives rise to the data in order to assess the generalized predictive utility of a proposed prediction rule. After learning a prediction rule based on a non-uniform sample, it is of interest to estimate the rule's error rate when applied to unobserved members of the population. Efron (1986) proposed a general class of covariance penalty inflated prediction error estimators that assume the available training data are representative of the target population for which the prediction rule is to be applied. We extend Efron's estimator to the complex sample context by incorporating Horvitz–Thompson sampling weights and show that it is consistent for the true generalization error rate when applied to the underlying superpopulation. The resulting Horvitz–Thompson–Efron estimator is equivalent to dAIC, a recent extension of Akaike's information criteria to survey sampling data, but is more widely applicable. The proposed methodology is assessed with simulations and is applied to models predicting renal function obtained from the large-scale National Health and Nutrition Examination Study survey. The Canadian Journal of Statistics 48: 204–221; 2020 © 2019 Statistical Society of Canada     

On a sampling scheme with inclusion probability proportional to size

A sampling scheme for selection of a sample of two units with inclusion probability proportionalto size is suggested which provides a non–negative variance estimator of the variance of Horvitz–Thompson estimator. The suggested sampling scheme is shown to perform better than many of the existing unequal probability and inclusion probability proportional to size sampling Achemes for a number of natural populations.     

A Monte Carlo comparison of some variance estimators of the Horvitz–Thompson estimator

At the design and estimation stage of a survey, large survey organization often uses auxiliary information. This article discusses various procedures for improving variance estimation of the Horvitz–Thompson estimator of a finite population total with the aid of auxiliary information. To study the design-based properties of the proposed variance estimators relative to the standard one, a small scale Monte Carlo study is performed.     

Location-shifts for improved estimation under Midzuno–Sen sampling scheme

Simple location-shifts for the study or auxiliary character are proposed under Midzuno–Sen sampling from a finite population. These aim at improving the efficiency of the classical Horvitz–Thompson estimator or the unbiased ratio estimator of a population total. It is demonstrated that the choice of the translation parameters is flexible. A few methods for assessing these parameters are outlined. The gain in efficiency of estimation is illustrated.     

An extension of Horvitz–Thompson estimator used in adaptive cluster sampling to continuous universe

In this paper, an extension of Horvitz–Thompson estimator used in adaptive cluster sampling to continuous universe is developed. Main new results are presented in theorems. The primary notions of discrete population are transferred to continuous population. First and second order inclusion probabilities for networks are delivered. Horvitz–Thompson estimator for adaptive cluster sampling in continuous universe is constructed. The unbiasedness of the estimator is proven. Variance and unbiased variance estimator are delivered. Finally, the theory is illustrated with an example.     

Efficient estimators for adaptive stratified sequential sampling

In stratified sampling, methods for the allocation of effort among strata usually rely on some measure of within-stratum variance. If we do not have enough information about these variances, adaptive allocation can be used. In adaptive allocation designs, surveys are conducted in two phases. Information from the first phase is used to allocate the remaining units among the strata in the second phase. Brown et al. [Adaptive two-stage sequential sampling, Popul. Ecol. 50 (2008), pp. 239–245] introduced an adaptive allocation sampling design – where the final sample size was random – and an unbiased estimator. Here, we derive an unbiased variance estimator for the design, and consider a related design where the final sample size is fixed. Having a fixed final sample size can make survey-planning easier. We introduce a biased Horvitz–Thompson type estimator and a biased sample mean type estimator for the sampling designs. We conduct two simulation studies on honey producers in Kurdistan and synthetic zirconium distribution in a region on the moon. Results show that the introduced estimators are more efficient than the available estimators for both variable and fixed sample size designs, and the conventional unbiased estimator of stratified simple random sampling design. In order to evaluate efficiencies of the introduced designs and their estimator furthermore, we first review some well-known adaptive allocation designs and compare their estimator with the introduced estimators. Simulation results show that the introduced estimators are more efficient than available estimators of these well-known adaptive allocation designs.     

Functional calibration estimation by the maximum entropy on the mean principle

We extend the problem of obtaining an estimator for the finite population mean parameter incorporating complete auxiliary information through calibration estimation in survey sampling under a functional data framework. The functional calibration sampling weights of the estimator are obtained by matching the calibration estimation problem with the maximum entropy on the mean – MEM – principle. In particular, the calibration estimation is viewed as an infinite-dimensional linear inverse problem following the structure of the MEM approach. We give a precise theoretical setting and estimate the functional calibration weights assuming, as prior measures, the centred Gaussian and compound Poisson random measures. Additionally, through a simple simulation study, we show that the proposed functional calibration estimator improves its accuracy compared with the Horvitz–Thompson one.     

A dual problem of calibration of design weights

In this note, a dual problem to the calibration of design weights of the Deville and Särndal [Calibration estimators in survey sampling, J. Amer. Statist. Assoc. 87 (1992), pp. 376–382] method has been considered. We conclude that the chi-squared distance between the design weights and the calibrated weights equals the square of the standardized Z-score obtained by the difference between the known population total of the auxiliary variable and its corresponding Horvitz and Thompson [A generalization of sampling without replacement from a finite universe, J. Amer. Statist. Assoc. 47 (1952), pp. 663–685] estimator divided by the sample standard deviation of the auxiliary variable to obtain the linear regression estimator in survey sampling.     

Sampling design proportional to order statistic of auxiliary variable

The sampling designs dependent on sample moments of auxiliary variables are well known. Lahiri (Bull Int Stat Inst 33:133–140, 1951) considered a sampling design proportionate to a sample mean of an auxiliary variable. Sing and Srivastava (Biometrika 67(1):205–209, 1980) proposed the sampling design proportionate to a sample variance while Wywiał (J Indian Stat Assoc 37:73–87, 1999) a sampling design proportionate to a sample generalized variance of auxiliary variables. Some other sampling designs dependent on moments of an auxiliary variable were considered e.g. in Wywiał (Some contributions to multivariate methods in, survey sampling. Katowice University of Economics, Katowice, 2003a); Stat Transit 4(5):779–798, 2000) where accuracy of some sampling strategies were compared, too.These sampling designs cannot be useful in the case when there are some censored observations of the auxiliary variable. Moreover, they can be much too sensitive to outliers observations. In these cases the sampling design proportionate to the order statistic of an auxiliary variable can be more useful. That is why such an unequal probability sampling design is proposed here. Its particular cases as well as its conditional version are considered, too. The sampling scheme implementing this sampling design is proposed. The inclusion probabilities of the first and second orders were evaluated. The well known Horvitz–Thompson estimator is taken into account. A ratio estimator dependent on an order statistic is constructed. It is similar to the well known ratio estimator based on the population and sample means. Moreover, it is an unbiased estimator of the population mean when the sample is drawn according to the proposed sampling design dependent on the appropriate order statistic.     

Semiparametric efficient estimation for the auxiliary outcome problem with the conditional mean model

The authors consider semiparametric efficient estimation of parameters in the conditional mean model for a simple incomplete data structure in which the outcome of interest is observed only for a random subset of subjects but covariates and surrogate (auxiliary) outcomes are observed for all. They use optimal estimating function theory to derive the semiparametric efficient score in closed form. They show that when covariates and auxiliary outcomes are discrete, a Horvitz‐Thompson type estimator with empirically estimated weights is semiparametric efficient. The authors give simulation studies validating the finite‐sample behaviour of the semiparametric efficient estimator and its asymptotic variance; they demonstrate the efficiency of the estimator in realistic settings.     

Double bootstrap estimation of variance under systematic sampling with probability proportional to size

Variance estimation under systematic sampling with probability proportional to size is known to be a difficult problem. We attempt to tackle this problem by the bootstrap resampling method. It is shown that the usual way to bootstrap fails to give satisfactory variance estimates. As a remedy, we propose a double bootstrap method which is based on certain working models and involves two levels of resampling. Unlike existing methods which deal exclusively with the Horvitz–Thompson estimator, the double bootstrap method can be used to estimate the variance of any statistic. We illustrate this within the context of both mean and median estimation. Empirical results based on five natural populations are encouraging.     

Variance approximation under balanced sampling

A balanced sampling design has the interesting property that Horvitz–Thompson estimators of totals for a set of balancing variables are equal to the totals we want to estimate, therefore the variance of Horvitz–Thompson estimators of variables of interest are reduced in function of their correlations with the balancing variables. Since it is hard to derive an analytic expression for the joint inclusion probabilities, we derive a general approximation of variance based on a residual technique. This approximation is useful even in the particular case of unequal probability sampling with fixed sample size. Finally, a set of numerical studies with an original methodology allows to validate this approximation.     

Bootstrapping probability-proportional-to-size samples via calibrated empirical population

A collection of six novel bootstrap algorithms, applied to probability-proportional-to-size samples, is explored for variance estimation, confidence interval and p-value production. Developed according to bootstrap fundamentals such as the mimicking principle and the plug-in rule, these algorithms make use of an empirical bootstrap population informed by sampled units each with assigned weight. Starting from the natural choice of Horvitz–Thompson (HT)-type weights, improvements based on calibration to known population features are fostered. Focusing on the population total as the parameter to be estimated and on the distribution of the HT estimator as the target of bootstrap estimation, simulation results are presented with the twofold objective of checking practical implementation and of investigating the statistical properties of the bootstrap estimates supplied by the algorithms explored.     

Variance estimation with Chao's sampling scheme

We show that the Hájek (Ann. Math Statist. (1964) 1491) variance estimator can be used to estimate the variance of the Horvitz–Thompson estimator when the Chao sampling scheme (Chao, Biometrika 69 (1982) 653) is implemented. This estimator is simple and can be implemented with any statistical packages. We consider a numerical and an analytic method to show that this estimator can be used. A series of simulations supports our findings.     

Inference on Polychotomous Responses in Finite Populations

Abstract. A model‐based predictive estimator is proposed for the population proportions of a polychotomous response variable, based on a sample from the population and on auxiliary variables, whose values are known for the entire population. The responses for the non‐sample units are predicted using a multinomial logit model, which is a parametric function of the auxiliary variables. A bootstrap estimator is proposed for the variance of the predictive estimator, its consistency is proved and its small sample performance is compared with that of an analytical estimator. The proposed predictive estimator is compared with other available estimators, including model‐assisted ones, both in a simulation study involving different sampling designs and model mis‐specification, and using real data from an opinion survey. The results indicate that the prediction approach appears to use auxiliary information more efficiently than the model‐assisted approach.