Theory and methods for partitioned Gini coefficients computed on post-stratified data

The Gini coefficient is used to measure inequality in populations. However, shifts in the population distribution may affect subgroups differently. Consequently, it can be informative to examine inequality separately for these segments. Consider an independently and identically distributed sample split based on ranking and compute the Gini coefficient for each partition. These coefficients, calculated from post-stratified data, are not functions of U-statistics. Therefore, previous theoretical and methodological results cannot be applied. In this article, the asymptotic joint distribution is derived for the partitioned coefficients and bootstrap methods for inference are developed. Finally, an application to per capita income across census tracts is examined.     

Unbiased estimation of income inequality

In a previous paper Gastwirth shows that a broad family of measures of inequality can be accurately estimated when the tax data are known in groups (more precisely, when we know the number of returns in each of several class intervals and their corresponding average income). In the present paper we show that some measures of the preceding family can be unbiasedly estimated when the tax data are individually known for a sample from the population. Specifically, we construct unbiased estimators of a particular measure of inequality in the samplings with and without replacement, and in the stratified samplings with and without replacement.     

Applications of a probability inequality to ranking and selection and other related problems

A probability inequality of conditionally independent and identically distributed (i.i.d.) random variables obtained recently by the author is applied to ranking and selection problems. It is shown that under both the indifference-zone and the subset formulations, the probability of a correct selection (PCS) is a cumulative probability of conditionally i.i.d, random variables. Therefore bounds on both the PCS and the sample size required can be obtained from that probability inequality. Applications of that inequality to other multiple decision problems are also considered. It is illustrated that general results concerning conditionally i.i.d. random variables are applicable to many problems in multiple decision theory.     

Limit theory of bivariate dual generalized order statistics with random index

The class of limit distribution functions of bivariate extreme, intermediate and central dual generalized order statistics from independent and identically distributed random variables with random sample size is fully characterized. Two cases are considered. The first case is when the random sample size is assumed to be independent of all basic random variables. The second case is when the interrelation of the random size and the basic random variables is not restricted.     

Sharp bounds for L-statistics from dependent samples of random length

Several distribution-free bounds on expected values of L-statistics based on the sample of possibly dependent and nonidentically distributed random variables are given in the case when the sample size is a random variable, possibly dependent on the observations, with values in the set {1,2,…}. Some bounds extend the results of Papadatos (2001a) to the case of random sample size. The others provide new evaluations even if the sample size is nonrandom. Some applications of the presented bounds are also indicated.     

Highest posterior density estimation from multiply censored Pareto data

In statistical practice, it is quite common that some data are unknown or disregarded for various reasons. In the present paper, on the basis of a multiply censored sample from a Pareto population, the problem of finding the highest posterior density (HPD) estimates of the inequality and precision parameters is discussed assuming a natural joint conjugate prior. HPD estimates are obtained in closed forms for complete or right censored data. In the general multiple censoring case, it is shown the existence and uniqueness of the estimates. Explicit lower and upper bounds are also provided. Due to the posterior unimodality, HPD credibility regions are simply connected sets. For illustration, two numerical examples are included.     

Bayesian tests on components of the compound symmetry covariance matrix

Complex dependency structures are often conditionally modeled, where random effects parameters are used to specify the natural heterogeneity in the population. When interest is focused on the dependency structure, inferences can be made from a complex covariance matrix using a marginal modeling approach. In this marginal modeling framework, testing covariance parameters is not a boundary problem. Bayesian tests on covariance parameter(s) of the compound symmetry structure are proposed assuming multivariate normally distributed observations. Innovative proper prior distributions are introduced for the covariance components such that the positive definiteness of the (compound symmetry) covariance matrix is ensured. Furthermore, it is shown that the proposed priors on the covariance parameters lead to a balanced Bayes factor, in case of testing an inequality constrained hypothesis. As an illustration, the proposed Bayes factor is used for testing (non-)invariant intra-class correlations across different group types (public and Catholic schools), using the 1982 High School and Beyond survey data.     

Multivariate Chebyshev Inequality With Estimated Mean and Variance

A variant of the well-known Chebyshev inequality for scalar random variables can be formulated in the case where the mean and variance are estimated from samples. In this article, we present a generalization of this result to multiple dimensions where the only requirement is that the samples are independent and identically distributed. Furthermore, we show that as the number of samples tends to infinity our inequality converges to the theoretical multi-dimensional Chebyshev bound.     

Finite population corrections for multivariate Bayes sampling

We consider the adjustment, based upon a sample of size n, of collections of vectors drawn from either an infinite or finite population. The vectors may be judged to be either normally distributed or, more generally, second-order exchangeable. We develop the work of Goldstein and Wooff (1998) to show how the familiar univariate finite population corrections (FPCs) naturally generalise to individual quantities in the multivariate population. The types of information we gain by sampling are identified with the orthogonal canonical variable directions derived from a generalised eigenvalue problem. These canonical directions share the same co-ordinate representation for all sample sizes and, for equally defined individuals, all population sizes enabling simple comparisons between both the effects of different sample sizes and of different population sizes. We conclude by considering how the FPC is modified for multivariate cluster sampling with exchangeable clusters. In univariate two-stage cluster sampling, we may decompose the variance of the population mean into the sum of the variance of cluster means and the variance of the cluster members within clusters. The first term has a FPC relating to the sampling fraction of clusters, the second term has a FPC relating to the sampling fraction of cluster size. We illustrate how this generalises in the multivariate case. We decompose the variance into two terms: the first relating to multivariate finite population sampling of clusters and the second to multivariate finite population sampling within clusters. We solve two generalised eigenvalue problems to show how to generalise the univariate to the multivariate: each of the two FPCs attaches to one, and only one, of the two eigenbases.     

An Adjustment to the Bartlett's Test for Small Sample Size

The Bartlett's test (1937) for equality of variances is based on the χ² distribution approximation. This approximation deteriorates either when the sample size is small (particularly < 4) or when the population number is large. According to a simulation investigation, we find a similar varying trend for the mean differences between empirical distributions of Bartlett's statistics and their χ² approximations. By using the mean differences to represent the distribution departures, a simple adjustment approach on the Bartlett's statistic is proposed on the basis of equal mean principle. The performance before and after adjustment is extensively investigated under equal and unequal sample sizes, with number of populations varying from 3 to 100. Compared with the traditional Bartlett's statistic, the adjusted statistic is distributed more closely to χ² distribution, for homogeneity samples from normal populations. The type I error is well controlled and the power is a little higher after adjustment. In conclusion, the adjustment has good control on the type I error and higher power, and thus is recommended for small samples and large population number when underlying distribution is normal.     

M-quantile models with application to poverty mapping

Over the last decade there has been growing demand for estimates of population characteristics at small area level. Unfortunately, cost constraints in the design of sample surveys lead to small sample sizes within these areas and as a result direct estimation, using only the survey data, is inappropriate since it yields estimates with unacceptable levels of precision. Small area models are designed to tackle the small sample size problem. The most popular class of models for small area estimation is random effects models that include random area effects to account for between area variations. However, such models also depend on strong distributional assumptions, require a formal specification of the random part of the model and do not easily allow for outlier robust inference. An alternative approach to small area estimation that is based on the use of M-quantile models was recently proposed by Chambers and Tzavidis (Biometrika 93(2):255–268, 2006) and Tzavidis and Chambers (Robust prediction of small area means and distributions. Working paper, 2007). Unlike traditional random effects models, M-quantile models do not depend on strong distributional assumption and automatically provide outlier robust inference. In this paper we illustrate for the first time how M-quantile models can be practically employed for deriving small area estimates of poverty and inequality. The methodology we propose improves the traditional poverty mapping methods in the following ways: (a) it enables the estimation of the distribution function of the study variable within the small area of interest both under an M-quantile and a random effects model, (b) it provides analytical, instead of empirical, estimation of the mean squared error of the M-quantile small area mean estimates and (c) it employs a robust to outliers estimation method. The methodology is applied to data from the 2002 Living Standards Measurement Survey (LSMS) in Albania for estimating (a) district level estimates of the incidence of poverty in Albania, (b) district level inequality measures and (c) the distribution function of household per-capita consumption expenditure in each district. Small area estimates of poverty and inequality show that the poorest Albanian districts are in the mountainous regions (north and north east) with the wealthiest districts, which are also linked with high levels of inequality, in the coastal (south west) and southern part of country. We discuss the practical advantages of our methodology and note the consistency of our results with results from previous studies. We further demonstrate the usefulness of the M-quantile estimation framework through design-based simulations based on two realistic survey data sets containing small area information and show that the M-quantile approach may be preferable when the aim is to estimate the small area distribution function.     

IS THE ASSUMPTION OF UNIFORM INTRA-CLASS CORRELATION EVER NEEDED?

When modelling a finite population it is sometimes assumed that the residuals from the regression model expectations are distributed with a uniform non-zero intra-class correlation. It is shown that if a certain vector is spanned by the columns of the design matrix (in the homoskedastic case this vector corresponds to the inclusion of a constant term) then such a model is underidentified and the assumption of a known non-zero correlation has almost no impact on the results of the regression analysis. When this vector is not spanned by the columns of the design matrix, a simpler alternative model can usually be fitted equally well to observations from any single population. The only exception occurs when the the intra-class correlation required is negative in sign.     

Skewness,kurtosis, and black-scholes option mispricing

The Black-Scholes option pricing model assumes that (instantaneous) common stock returns are normally distributed. However, the observed distribution exhibits deviations from normality; in particular skewness and kurtosis. We attribute these deviations to gross data errors. Using options' transactions data, we establish that the sample standard deviation, sample skewness, and sample kurtosis contribute to the Black-Scholes model's observed mispricing of a sample from the Berkeley Options Data Base of 2323 call options written on 88 common stocks paying no dividends during the options'life. Following Huber's statement that the primary case for robust statistics is when the shape of the observed distribution deviates slightly from the assumed distribution (usually the Gaussian), we show that robust volatility estimators eliminate the mispricing with respect to sample skewness and sample kurtosis, and significantly improve the Black-Scholes model's pricing performance with respect to estimated volatility.     

LIMIT THEOREMS FOR PROPORTIONS OF OBSERVATIONS FALLING INTO RANDOM REGIONS DETERMINED BY ORDER STATISTICS

In this paper, we study asymptotic behavior of proportions of sample observations that fall into random regions determined by a given Borel set and an order statistic. We show that these proportions converge almost surely to some population quantities as the sample size increases to infinity. We derive our results for independent and identically distributed observations from an arbitrary cumulative distribution function, in particular, we allow samples drawn from discontinuous laws. We also give extensions of these results to the case of randomly indexed samples with some dependence between observations.     

A power study of goodness-of-fit tests for multivariate normality implemented in R

Multivariate statistical analysis procedures often require data to be multivariate normally distributed. Many tests have been developed to verify if a sample could indeed have come from a normally distributed population. These tests do not all share the same sensitivity for detecting departures from normality, and thus a choice of test is of central importance. This study investigates through simulated data the power of those tests for multivariate normality implemented in the statistic software R and pits them against the variant of testing each marginal distribution for normality. The results of testing two-dimensional data at a level of significance α=5% showed that almost one-third of those tests implemented in R do not have a type I error below this. Other tests outperformed the naive variant in terms of power even when the marginals were not normally distributed. Even though no test was consistently better than all alternatives with every alternative distribution, the energy-statistic test always showed relatively good power across all tested sample sizes.     

Comparison of combinatoric and likelihood ratio procedures for classifying samples

A random sample is to be classified as coming from one of two normally distributed populations with known parameters. Combinatoric procedures which classify the sample based upon the sample mean(s) and variance(s) are described for the univariate and multivariate problems. Comparisons of misclassification probabilities are made between the combinatoric and the likelihood ratio procedure in the univariate case and between two alternative combinatoric procedures in the bivariate case.     

The problem of outlier tests in sample surveys

Outliers can occur as readily in samples from the finite populations (e.g. in sample surveys) as in samples from infinite populations. However, in the vast literature on outliers there is almost no mention of outlier tests for data from sample surveys. We examine the behaviour of some standard outlier test statistics for infinite populations when these are applied to finite populations, examining their properties by extensive simulation studies. Some anomalous results are obtained Nsuggesting a fundamental difficulty in testing outliers for the finite population case.     

Estimating and testing generalized linear models under inequality restrictions

We consider maximum likelihood estimation and likelihood ratio tests under inequality restrictions on the parameters. A special case are order restrictions, which may appear for example in connection with effects of an ordinal qualitative covariate. Our estimation approach is based on the principle of sequential quadratic programming, where the restricted estimate is computed iteratively and a quadratic optimization problem under inequality restrictions is solved in each iteration. Testing for inequality restrictions is based on the likelihood ratio principle. Under certain regularity assumptions the likelihood ratio test statistic is asymptotically distributed like a mixture of χ², where the weights are a function of the restrictions and the information matrix. A major problem in theory is that in general there is no unique least favourable point. We present some empirical findings on finite-sample behaviour of tests and apply the methods to examples from credit scoring and dentistry.     

Some Aspects of Probability of Correct Selections

In an experiment of treatment selections, random samples are drawn from k populations with ordered means. The probability that a sample statistic from the population with the highest mean turns out to be ranked the highest is referred to as the probability of correct selection (PCS). An inequality was proved previously that shows the monotonicity of PCS with respect to change in variance of the samples. In this article, we first present a more general form of the probability inequality to be used to investigate PCS. An extension of the monotonicity of PCS to order statistics is considered. We show that the PCS of the smallest order statistic preserves the monotonicity. Additionally, a normal approximation method is used to further generalize the theory. The general order statistics will not enjoy the same properties, as we reveal the obstacles, and a numerical counter example.