Assessing the impact of initial nonresponse and attrition in the analysis of unemployment duration with panel surveys

We show how register data combined at person-level with survey data can be used to conduct a novel type of nonresponse analysis in a panel survey. The availability of register data provides a unique opportunity to directly test the type of the missingness mechanism as well as estimate the size of bias due to initial nonresponse and attrition. We are also able to study in-depth the determinants of initial nonresponse and attrition. We use the Finnish subset of the European Community Household Panel (FI ECHP) data combined with register panel data and unemployment spells as outcome variables of interest. Our results show that initial nonresponse and attrition are clearly different processes driven by different background variables. Both the initial nonresponse and attrition mechanisms are nonignorable with respect to analysis of unemployment spells. Finally, our results suggest that initial nonresponse may play a role at least as important as attrition in causing bias. This result challenges the common view of attrition being the main threat to the value of panel data.     

含非随机缺失数据的面板数据参数估计方法

抽样调查领域常采用对多个受访者进行跟踪调查得到面板数据,进而对总体特性进行统计推断,在面板数据中常含缺失数据,大多数处理面板缺失数据的软件都是直接删去含缺失值的受访者以得到完全数据集,当数据缺失机制为非随机缺失时会导致总体参数估计结果有偏。本文针对数据缺失机制为非随机缺失情形下,如何对面板数据进行统计分析进行了阐述,主要采用的是基于模型的似然推断法,对目标变量、缺失指示变量和随机效应向量的联合分布建模,在已有选择模型和模式混合模型的基础上,引入随机效应,研究目标变量期望的计算方法,并研究随机效应杂合模型下参数的估计方法,在变量分布相对简单的情形下给出了用极大似然法推断总体参数的估计步骤,最后通过模拟分析比较方法的优劣。     

Bias-Corrected Estimation in Dynamic Panel Data Models

This study develops a new bias-corrected estimator for the fixed-effects dynamic panel data model and derives its limiting distribution for finite number of time periods, T, and large number of cross-section units, N. The bias-corrected estimator is derived as a bias correction of the least squares dummy variable (within) estimator. It does not share some of the drawbacks of recently developed instrumental variables and generalized method-of-moments estimators and is relatively easy to compute. Monte Carlo experiments provide evidence that the bias-corrected estimator performs well even in small samples. The proposed technique is applied in an empirical analysis of unemployment dynamics at the U.S. state level for the 1991–2000 period.     

Using Auxiliary Data for Binomial Parameter Estimation with Nonignorable Nonresponse

Nonignorable nonresponse is a nonresponse mechanism that depends on the values of the variable having nonresponse. When an observed data of a binomial distribution suffer missing values from a nonignorable nonresponse mechanism, the binomial distribution parameters become unidentifiable without any other auxiliary information or assumption. To address the problems of non identifiability, existing methods mostly based on the log-linear regression model. In this article, we focus on the model when the nonresponse is nonignorable and we consider to use the auxiliary data to improve identifiability; furthermore, we derive the maximum likelihood estimator (MLE) for the binomial proportion and its associated variance. We present results for an analysis of real-life data from the SARS study in China. Finally, the simulation study shows that the proposed method gives promising results.     

Pseudo Panel Data Models With Cohort Interactive Effects

When genuine panel data samples are not available, repeated cross-sectional surveys can be used to form so-called pseudo panels. In this article, we investigate the properties of linear pseudo panel data estimators with fixed number of cohorts and time observations. We extend standard linear pseudo panel data setup to models with factor residuals by adapting the quasi-differencing approach developed for genuine panels. In a Monte Carlo study, we find that the proposed procedure has good finite sample properties in situations with endogeneity, cohort interactive effects, and near nonidentification. Finally, as an illustration the proposed method is applied to data from Ecuador to study labor supply elasticity. Supplementary materials for this article are available online.     

Modeling Nonignorable Nonresponse in Categorical Panel Data With an Example in Estimating Gross Labor-Force Flows

Many large-scale sample surveys use panel designs under which sampled individuals are interviewed several times before being dropped from the sample. The longitudinal data bases available from such surveys could be used to provide estimates of gross change over time. One problem in using these data to estimate gross change is how to handle the period-to-period nonresponse. This nonresponse is typically nonrandom and, furthermore, may be nonignorable in that it cannot be accounted for by other observed quantities in the data. Under the models proposed in this article, which are appropriate for the analysis of categorical data, the probability of nonresponse may be taken to be a function of the missing variable of interest. The proposed models are fit using maximum likelihood estimation. As an example, the method is applied to the problem of estimating gross flows in labor-force participation using data from the Current Population Survey and the Canadian Labour Force Survey.     

Bayesian estimation of the multidimensional graded response model with nonignorable missing data

A Bayesian approach is developed for analysing item response models with nonignorable missing data. The relevant model for the observed data is estimated concurrently in conjunction with the item response model for the missing-data process. Since the approach is fully Bayesian, it can be easily generalized to more complicated and realistic models, such as those models with covariates. Furthermore, the proposed approach is illustrated with item response data modelled as the multidimensional graded response models. Finally, a simulation study is conducted to assess the extent to which the bias caused by ignoring the missing-data mechanism can be reduced.     

Weighted composite quantile regression analysis for nonignorable missing data using nonresponse instrument

Efficient statistical inference on nonignorable missing data is a challenging problem. This paper proposes a new estimation procedure based on composite quantile regression (CQR) for linear regression models with nonignorable missing data, that is applicable even with high-dimensional covariates. A parametric model is assumed for modelling response probability, which is estimated by the empirical likelihood approach. Local identifiability of the proposed strategy is guaranteed on the basis of an instrumental variable approach. A set of data-based adaptive weights constructed via an empirical likelihood method is used to weight CQR functions. The proposed method is resistant to heavy-tailed errors or outliers in the response. An adaptive penalisation method for variable selection is proposed to achieve sparsity with high-dimensional covariates. Limiting distributions of the proposed estimators are derived. Simulation studies are conducted to investigate the finite sample performance of the proposed methodologies. An application to the ACTG 175 data is analysed.     

Provincial Conditional Income Convergence in China, 1953-1997: A Panel Data Approach

This paper examines the tendency towards income convergence among China's main provinces during the two periods: the pre-reform period 1953-1977 and the reform period 1978-1997 using the framework of the Solow growth model. The panel data method accounts for not only province-specific initial technology level but also the heterogeneity of the technological progress rate between the fast-growing coastal and interior provinces. Estimation problems of weak instruments and endogeneity are addressed by the use of a system generalized method of moments (GMM) estimator. The main empirical finding is that there is a system-wide income divergence during the reform period because the coastal provinces do not share a common technology progress rate with the interior provinces.     

Darling–Erdős limit results for change-point detection in panel data

We wish to test the null hypothesis if the means of N panels remain the same during the observation period of length T. A quasi-likelihood argument leads to self-normalized statistics whose limit distribution under the null hypothesis is double exponential. The main results are derived assuming that the each panel is based on independent observations and then extended to linear processes. The proofs are based on an approximation of the sum of squared CUSUM processes using the Skorokhod embedding scheme. A simulation study illustrates that our results can be used in case of small and moderate N and T. We apply our results to detect change in the “corruption index”.     

Bayesian Analysis of Nonlinear Reproductive Dispersion Mixed Models for Longitudinal Data with Nonignorable Missing Covariates

This article proposes a Bayesian approach, which can simultaneously obtain the Bayesian estimates of unknown parameters and random effects, to analyze nonlinear reproductive dispersion mixed models (NRDMMs) for longitudinal data with nonignorable missing covariates and responses. The logistic regression model is employed to model the missing data mechanisms for missing covariates and responses. A hybrid sampling procedure combining the Gibber sampler and the Metropolis-Hastings algorithm is presented to draw observations from the conditional distributions. Because missing data mechanism is not testable, we develop the logarithm of the pseudo-marginal likelihood, deviance information criterion, the Bayes factor, and the pseudo-Bayes factor to compare several competing missing data mechanism models in the current considered NRDMMs with nonignorable missing covaraites and responses. Three simulation studies and a real example taken from the paediatric AIDS clinical trial group ACTG are used to illustrate the proposed methodologies. Empirical results show that our proposed methods are effective in selecting missing data mechanism models.     

Estimation with unbalanced panel data having covariate measurement error

Panel data with covariate measurement error appear frequently in various studies. Due to the sampling design and/or missing data, panel data are often unbalanced in the sense that panels have different sizes. For balanced panel data (i.e., panels having the same size), there exists a generalized method of moments (GMM) approach for adjusting covariate measurement error, which does not require additional validation data. This paper extends the GMM approach of adjusting covariate measurement error to unbalanced panel data. Two health related longitudinal surveys are used to illustrate the implementation of the proposed method.     

TIMING OF SURVEYS FOR PUBLIC POLICY

This keynote address at the 7th Australian Statistical Conference (1984) discusses briefly seven modifications of sample design for improving the usefulness and timeliness of surveys relevant to public poky. These are: better estimates for small domains, cumulating rolling samples, more panel surveys, multipurpose designs for periodic samples, split panel designs (SPD) combining panels with nonoverlapping samples, and frequent collections cumulated for less frequent reporting periods.     

A hierarchical Bayesian model for binary data incorporating selection bias

We consider a Bayesian nonignorable model to accommodate a nonignorable selection mechanism for predicting small area proportions. Our main objective is to extend a model on selection bias in a previously published paper, coauthored by four authors, to accommodate small areas. These authors assume that the survey weights (or their reciprocals that we also call selection probabilities) are available, but there is no simple relation between the binary responses and the selection probabilities. To capture the nonignorable selection bias within each area, they assume that the binary responses and the selection probabilities are correlated. To accommodate the small areas, we extend their model to a hierarchical Bayesian nonignorable model and we use Markov chain Monte Carlo methods to fit it. We illustrate our methodology using a numerical example obtained from data on activity limitation in the U.S. National Health Interview Survey. We also perform a simulation study to assess the effect of the correlation between the binary responses and the selection probabilities.     

Random Effects Modeling and the Zero-Inflated Poisson Distribution

Overdispersion due to a large proportion of zero observations in data sets is a common occurrence in many applications of many fields of research; we consider such scenarios in count panel (longitudinal) data. A well-known and widely implemented technique for handling such data is that of random effects modeling, which addresses the serial correlation inherent in panel data, as well as overdispersion. To deal with the excess zeros, a zero-inflated Poisson distribution has come to be canonical, which relaxes the equal mean-variance specification of a traditional Poisson model and allows for the larger variance characteristic of overdispersed data. A natural proposal then to approach count panel data with overdispersion due to excess zeros is to combine these two methodologies, deriving a likelihood from the resulting conditional probability. In performing simulation studies, we find that this approach in fact poses problems of identifiability. In this article, we construct and explain in full detail why a model obtained from the marriage of two classical and well-established techniques is unidentifiable and provide results of simulation studies demonstrating this effect. A discussion on alternative methodologies to resolve the problem is provided in the conclusion.     

Diagnostics for the diagnostics

Abstract

This paper examines the tendency towards income convergence among China's main provinces during the two periods: the pre‐reform period 1953–1977 and the reform period 1978–1997 using the framework of the Solow growth model. The panel data method accounts for not only province‐specific initial technology level but also the heterogeneity of the technological progress rate between the fast‐growing coastal and interior provinces. Estimation problems of weak instruments and endogeneity are addressed by the use of a system generalized method of moments (GMM) estimator. The main empirical finding is that there is a system‐wide income divergence during the reform period because the coastal provinces do not share a common technology progress rate with the interior provinces.     

A residual-based test of the null of cointegration in panel data

This paper proposes a residual-based Lagrange Multiplier (LM) test for the null of cointegration in panel data. The test is analogous to the locally best unbiased invariant (LBUI) for a moving average (MA) unit root. The asymptotic distribution of the test is derived under the null. Monte Carlo simulations are performed to study the size and power properties of the proposed test.

overall, the empirical sizes of the LM-FM and LM-DOLs are close to the true size even in small samples. The power is quite good for the panels where T ≥ 50, and decent with panels for fewer observation in T. In our fixed sample of N = 50 and T = 50, the presence of a moving average and correlation between the LM-DOLS test seems to be better at correcting these effects, although in some cases the LM-FM test is more powerful.

Although much of the non-stationary time series econometrics has been criticized for having more to do with the specific properties of the data set rather than underlying economic models, the recent development of the cointegration literature has allowed for a concrete bridge between economic long run theory and time series methods. Our test now allows for the testing of the null of cointegration in a panel setting and should be of considerable interest to economists in a wide variety of fields.     

A Test for Slope Heterogeneity in Fixed Effects Models

Typical panel data models make use of the assumption that the regression parameters are the same for each individual cross-sectional unit. We propose tests for slope heterogeneity in panel data models. Our tests are based on the conditional Gaussian likelihood function in order to avoid the incidental parameters problem induced by the inclusion of individual fixed effects for each cross-sectional unit. We derive the Conditional Lagrange Multiplier test that is valid in cases where N → ∞ and T is fixed. The test applies to both balanced and unbalanced panels. We expand the test to account for general heteroskedasticity where each cross-sectional unit has its own form of heteroskedasticity. The modification is possible if T is large enough to estimate regression coefficients for each cross-sectional unit by using the MINQUE unbiased estimator for regression variances under heteroskedasticity. All versions of the test have a standard Normal distribution under general assumptions on the error distribution as N → ∞. A Monte Carlo experiment shows that the test has very good size properties under all specifications considered, including heteroskedastic errors. In addition, power of our test is very good relative to existing tests, particularly when T is not large.     

Constrained inference for generalized linear models with incomplete covariate data

Missing data are common in many experiments, including surveys, clinical trials, epidemiological studies, and environmental studies. Unconstrained likelihood inferences for generalized linear models (GLMs) with nonignorable missing covariates have been studied extensively in the literature. However, parameter orderings or constraints may occur naturally in practice, and thus the efficiency of a statistical method may be improved by incorporating parameter constraints into the likelihood function. In this paper, we consider constrained inference for analysing GLMs with nonignorable missing covariates under linear inequality constraints on the model parameters. Specifically, constrained maximum likelihood (ML) estimation is based on the gradient projection expectation maximization approach. Further, we investigate the asymptotic null distribution of the constrained likelihood ratio test (LRT). Simulations study the empirical properties of the constrained ML estimators and LRTs, which demonstrate improved precision of these constrained techniques. An application to contaminant levels in an environmental study is also presented.