On the planning and design of sample surveys

Surveys rely on structured questions used to map out reality, using sample observations from a population frame, into data that can be statistically analyzed. This paper focuses on the planning and design of surveys, making a distinction between individual surveys, household surveys and establishment surveys. Knowledge from cognitive science is used to provide guidelines on questionnaire design. Non-standard, but simple, statistical methods are described for analyzing survey results. The paper is based on experience gained by conducting over 150 customer satisfaction surveys in Europe, America and the Far East.     

An Evaluation of a Large-scale Cati Household Survey Using Random Digit Dialling

Computer-assisted telephone interviewing and random digit dialling are increasingly being used to conduct household surveys in Australia. However, there is little published information concerning Australian experience with such surveys. In 1995 the Government Statistician's Office in Queensland conducted a household survey to study population migration using these techniques. The survey involved a sample of 110 000 telephone numbers resulting in 38 000 responding households. This article describes a computerized survey management system that was developed and which provided information concerning important operational and quality aspects of the survey.     

Regression Coefficient Estimation in Dual Frame Surveys

Dual frame surveys, in which independent samples are selected from two frames to decrease survey costs or to improve coverage, can present challenges for regression coefficient estimation because of complex designs and unknown degree of overlap. In this research, we developed four regression coefficient estimators in dual frame surveys. Simulation results show that all the proposed methods work well.     

QUALITY ISSUES IN TELEPHONE SURVEYS: COVERAGE, NON-RESPONSE and QUOTA SAMPLING

The quality of a telephone survey is affected by several factors: telephone coverage, non-response, the methods used to select households and persons, and the quality of responses obtained from respondents. Data are provided which show that a large proportion of Australian households have telephone connections. However, telephone coverage is not uniform and some subgroups of the population have much lower connection rates. This paper reviews evidence of the effect of non-response and the effectiveness of repeated call backs, and reports the results of a new study. The use of quota sampling to select respondents from randomly selected households is also examined. The results suggest that telephone surveys under-represent older persons and the unemployed, and over-represent middle-aged persons. It is shown that while call backs can increase the response rate, the effect on the composition of the sample and resulting estimates is minimal. The main effects are due to refusals and variation in coverage rates.     

Responsive design for household surveys: tools for actively controlling survey errors and costs

Summary.  Over the past few years surveys have expanded to new populations, have incorporated measurement of new and more complex substantive issues and have adopted new data collection tools. At the same time there has been a growing reluctance among many household populations to participate in surveys. These factors have combined to present survey designers and survey researchers with increased uncertainty about the performance of any given survey design at any particular point in time. This uncertainty has, in turn, challenged the survey practitioner's ability to control the cost of data collection and quality of resulting statistics. The development of computer-assisted methods for data collection has provided survey researchers with tools to capture a variety of process data ('paradata') that can be used to inform cost–quality trade-off decisions in realtime. The ability to monitor continually the streams of process data and survey data creates the opportunity to alter the design during the course of data collection to improve survey cost efficiency and to achieve more precise, less biased estimates. We label such surveys as 'responsive designs'. The paper defines responsive design and uses examples to illustrate the responsive use of paradata to guide mid-survey decisions affecting the non-response, measurement and sampling variance properties of resulting statistics.     

Simulation and quality of a synthetic close-to-reality employer–employee population

It is of essential importance that researchers have access to linked employer–employee data, but such data sets are rarely available for researchers or the public. Even in case that survey data have been made available, the evaluation of estimation methods is usually done by complex design-based simulation studies. For this aim, data on population level are needed to know the true parameters that are compared with the estimations derived from complex samples. These samples are usually drawn from the population under various sampling designs, missing values and outlier scenarios. The structural earnings statistics sample survey proposes accurate and harmonized data on the level and structure of remuneration of employees, their individual characteristics and the enterprise or place of employment to which they belong in EU member states and candidate countries. At the basis of this data set, we show how to simulate a synthetic close-to-reality population representing the employer and employee structure of Austria. The proposed simulation is based on work of A. Alfons, S. Kraft, M. Templ, and P. Filzmoser [{\em On the simulation of complex universes in the case of applying the German microcensus}, DACSEIS research paper series No. 4, University of Tübingen, 2003] and R. Münnich and J. Schürle [{\em Simulation of close-to-reality population data for household surveys with application to EU-SILC}, Statistical Methods & Applications 20(3) (2011c), pp. 383–407]. However, new challenges are related to consider the special structure of employer–employee data and the complexity induced with the underlying two-stage design of the survey. By using quality measures in form of simple summary statistics, benchmarking indicators and visualizations, the simulated population is analysed and evaluated. An accompanying study on literature has been made to select the most important benchmarking indicators.     

Simulation of close-to-reality population data for household surveys with application to EU-SILC

Statistical simulation in survey statistics is usually based on repeatedly drawing samples from population data. Furthermore, population data may be used in courses on survey statistics to explain issues regarding, e.g., sampling designs. Since the availability of real population data is in general very limited, it is necessary to generate synthetic data for such applications. The simulated data need to be as realistic as possible, while at the same time ensuring data confidentiality. This paper proposes a method for generating close-to-reality population data for complex household surveys. The procedure consists of four steps for setting up the household structure, simulating categorical variables, simulating continuous variables and splitting continuous variables into different components. It is not required to perform all four steps so that the framework is applicable to a broad class of surveys. In addition, the proposed method is evaluated in an application to the European Union Statistics on Income and Living Conditions (EU-SILC).     

Embedded experiments in repeated and overlapping surveys

Summary.  Statistical agencies make changes to the data collection methodology of their surveys to improve the quality of the data collected or to improve the efficiency with which they are collected. For reasons of cost it may not be possible to estimate the effect of such a change on survey estimates or response rates reliably, without conducting an experiment that is embedded in the survey which involves enumerating some respondents by using the new method and some under the existing method. Embedded experiments are often designed for repeated and overlapping surveys; however, previous methods use sample data from only one occasion. The paper focuses on estimating the effect of a methodological change on estimates in the case of repeated surveys with overlapping samples from several occasions. Efficient design of an embedded experiment that covers more than one time point is also mentioned. All inference is unbiased over an assumed measurement model, the experimental design and the complex sample design. Other benefits of the approach proposed include the following: it exploits the correlation between the samples on each occasion to improve estimates of treatment effects; treatment effects are allowed to vary over time; it is robust against incorrectly rejecting the null hypothesis of no treatment effect; it allows a wide set of alternative experimental designs. This paper applies the methodology proposed to the Australian Labour Force Survey to measure the effect of replacing pen-and-paper interviewing with computer-assisted interviewing. This application considered alternative experimental designs in terms of their statistical efficiency and their risks to maintaining a consistent series. The approach proposed is significantly more efficient than using only 1 month of sample data in estimation.     

Imputation of Household Survey Data Using Linear Mixed Models

Mixed models are regularly used in the analysis of clustered data, but are only recently being used for imputation of missing data. In household surveys where multiple people are selected from each household, imputation of missing values should preserve the structure pertaining to people within households and should not artificially change the apparent intracluster correlation (ICC). This paper focuses on the use of multilevel models for imputation of missing data in household surveys. In particular, the performance of a best linear unbiased predictor for both stochastic and deterministic imputation using a linear mixed model is compared to imputation based on a single level linear model, both with and without information about household respondents. In this paper an evaluation is carried out in the context of imputing hourly wage rate in the Household, Income and Labour Dynamics of Australia Survey. Nonresponse is generated under various assumptions about the missingness mechanism for persons and households, and with low, moderate and high intra‐household correlation to assess the benefits of the multilevel imputation model under different conditions. The mixed model and single level model with information about the household respondent lead to clear improvements when the ICC is moderate or high, and when there is informative missingness.     

Fitting disposition codes to mobile phone surveys: experiences from studies in Finland, Slovenia and the USA

Summary.  Using mobile phones to conduct survey interviews has gathered momentum recently. However, using mobile telephones in surveys poses many new challenges. One important challenge involves properly classifying final case dispositions to understand response rates and non-response error and to implement responsive survey designs. Both purposes demand accurate assessments of the outcomes of individual call attempts. By looking at actual practices across three countries, we suggest how the disposition codes of the American Association for Public Opinion Research, which have been developed for telephone surveys, can be modified to fit mobile phones. Adding an international dimension to these standard definitions will improve survey methods by making systematic comparisons across different contexts possible.     

On randomized response surveys for estimating a proportion

In studies about sensitive characteristics, randomized response (RR) methods are useful for generating reliable data, protecting respondents’ privacy. It is shown that all RR surveys for estimating a proportion can be encompassed in a common model and some general results for statistical inferences can be used for any given survey. The concepts of design and scheme are introduced for characterizing RR surveys. Some consequences of comparing RR designs based on statistical measures of efficiency and respondent’ protection are discussed. In particular, such comparisons lead to the designs that may not be suitable in practice. It is suggested that one should consider other criteria and the scheme parameters for planning a RR survey.     

Enriching Surveys with Supplementary Data and its Application to Studying Wage Regression

We consider the problem of supplementing survey data with additional information from a population. The framework we use is very general; examples are missing data problems, measurement error models and combining data from multiple surveys. We do not require the survey data to be a simple random sample of the population of interest. The key assumption we make is that there exists a set of common variables between the survey and the supplementary data. Thus, the supplementary data serve the dual role of providing adjustments to the survey data for model consistencies and also enriching the survey data for improved efficiency. We propose a semi‐parametric approach using empirical likelihood to combine data from the two sources. The method possesses favourable large and moderate sample properties. We use the method to investigate wage regression using data from the National Longitudinal Survey of Youth Study.     

Surveying migrant households: a comparison of census-based, snowball and intercept point surveys

Summary.  Few representative surveys of households of migrants exist, limiting our ability to study the effects of international migration on sending families. We report the results of an experiment that was designed to compare the performance of three alternative survey methods in collecting data from Japanese–Brazilian families, many of whom send migrants to Japan. The three surveys that were conducted were households selected randomly from a door-to-door listing using the Brazilian census to select census blocks, a snowball survey using Nikkei community groups to select the seeds and an intercept point survey that was collected at Nikkei community gatherings, ethnic grocery stores, sports clubs and other locations where family members of migrants are likely to congregate. We analyse how closely well-designed snowball and intercept point surveys can approach the much more expensive census-based method in terms of giving information on the characteristics of migrants, the level of remittances received and the incidence and determinants of return migration.     

On the Problem of Compromise Allocation in Multi-Response Stratified Sample Surveys

In stratified sample surveys, the problem of determining the optimum allocation is well known due to articles published in 1923 by Tschuprow and in 1934 by Neyman. The articles suggest the optimum sample sizes to be selected from each stratum for which sampling variance of the estimator is minimum for fixed total cost of the survey or the cost is minimum for a fixed precision of the estimator. If in a sample survey more than one characteristic is to be measured on each selected unit of the sample, that is, the survey is a multi-response survey, then the problem of determining the optimum sample sizes to various strata becomes more complex because of the non-availability of a single optimality criterion that suits all the characteristics. Many authors discussed compromise criterion that provides a compromise allocation, which is optimum for all characteristics, at least in some sense. Almost all of these authors worked out the compromise allocation by minimizing some function of the sampling variances of the estimators under a single cost constraint. A serious objection to this approach is that the variances are not unit free so that minimizing any function of variances may not be an appropriate objective to obtain a compromise allocation. This fact suggests the use of coefficient of variations instead of variances. In the present article, the problem of compromise allocation is formulated as a multi-objective non-linear programming problem. By linearizing the non-linear objective functions at their individual optima, the problem is approximated to an integer linear programming problem. Goal programming technique is then used to obtain a solution to the approximated problem.     

书面调查和网络调查的区别——两种数据收集方法的比较

随着网络调查的兴起,研究者必须确认网络调查与传统的纸笔调查效果是否相同。从数据收集质量和测量效果两个方面对纸笔调查与网络调查进行了比较。研究发现：纸笔调查与网络调查的测量模型和测量信度没有显著差异,但是,纸笔调查的测量均值高于网络调查,而网络调查的测量误差高于纸笔调查,网络调查的缺失率更低,纸笔调查与网络调查具有测量不变性。     

网络调查中的非抽样误差及其预防措施

互联网的迅速发展,给统计调查方法带来了巨大的影响,在互联网的基础之上发展起来的网络调查方法,以其独特的优势,日益受到人们的青睐。本文结合调查误差分析的理论,根据网络自身的特点,分析了网络调查的非抽样误差的来源,并提出了减少误差的方法。     

Multilevel modelling of refusal and non-contact in household surveys: evidence from six UK Government surveys

Summary.  We analyse household unit non-response in six major UK Government surveys by using a multilevel multinomial modelling approach. The models are guided by current conceptual frameworks and theories of survey participation. One key feature of the analysis is the investigation of the extent to which effects of household characteristics are survey specific. The analysis is based on the 2001 UK Census Link Study, which is a unique source of data containing an unusually rich set of auxiliary variables. The study contains the response outcome of six surveys, linked to census data and interviewer observations for both respondents and non-respondents.     

Statistical problems in multi-variate (etiological) surveys

A number of multi-variate etiological surveys are analyzed for recurrent sources of bias in balanced and purposive sampling designs. Three nonsampling components emerge that may dominate the total error of a sample survey estimate. Outstanding among these appear to be administrative consideration of cost and convenience which may actually determine a sampling procedure, especially by reliance on voluntary participation, proxy responses and case-finding methods that restrict the sample. Next is lack of comparability of population series that differ on some initial state (as to smoking). Finally, errors are caused by strong beliefs in what results should be. Extensive experience has now shown that it may not be possible to conduct a satisfactory etiological inquiry by use of surveys using nonrandom population samples.     

住户调查中代表性样本的一种探索获取方法——平衡抽样设计

住户调查是我国社会经济统计调查体系的重要组成部分,样本代表性直接决定统计数据质量。多阶段抽样中初级单元的方差对估计的影响是主要的,因此本文结合2010年全国第六次人口普查分县数据,采用平衡抽样设计获取初级单元的代表性样本-平衡样本。对代表性样本的事后评估结果表明,样本结构与总体结构吻合,目标估计的误差很小,说明了本文平衡设计的有效性。     

On Sample Allocation in Multivariate Surveys

The problem of a sample allocation between strata in the case of multiparameter surveys is considered in this article. There are several multivariate sample allocation methods and, moreover, several criteria to deal with in such a case. A maximum coefficient of variation of estimators of the population mean of characters under study is taken as the optimality criterion. This article contains a study on a group of the methods that are easy to implement and do not need complex numerical computation; however, they all are approximate. Five such methods are presented and compared using a simulation study. Finally, it is shown which methods should be considered when designing a survey in which the multivariate sample allocation is to be involved.