Measuring the complexity of generalized linear hierarchical models

Measuring a statistical model's complexity is important for model criticism and comparison. However, it is unclear how to do this for hierarchical models due to uncertainty about how to count the random effects. The authors develop a complexity measure for generalized linear hierarchical models based on linear model theory. They demonstrate the new measure for binomial and Poisson observables modeled using various hierarchical structures, including a longitudinal model and an areal‐data model having both spatial clustering and pure heterogeneity random effects. They compare their new measure to a Bayesian index of model complexity, the effective number p_D of parameters (Spiegelhalter, Best, Carlin & van der Linde 2002); the comparisons are made in the binomial and Poisson cases via simulation and two real data examples. The two measures are usually close, but differ markedly in some instances where p_D is arguably inappropriate. Finally, the authors show how the new measure can be used to approach the difficult task of specifying prior distributions for variance components, and in the process cast further doubt on the commonly‐used vague inverse gamma prior.     

On hypothesis tests in misspecified change-point problems for a Poisson process

Consider an inhomogeneous Poisson process X on [0, T] whose unk-nown intensity function “switches” from a lower function g_* to an upper function h_* at some unknown point ?_* that has to be identified. We consider two known continuous functions g and h such that g_*(t) ? g(t) < h(t) ? h_*(t) for 0 ? t ? T. We describe the behavior of the generalized likelihood ratio and Wald’s tests constructed on the basis of a misspecified model in the asymptotics of large samples. The power functions are studied under local alternatives and compared numerically with help of simulations. We also show the following robustness result: the Type I error rate is preserved even though a misspecified model is used to construct tests.     

Modeling with Mixtures of Linear Regressions

Consider data (x ₁,y ₁),...,(x _n,y _n), where each x _i may be vector valued, and the distribution of y _i given x _i is a mixture of linear regressions. This provides a generalization of mixture models which do not include covariates in the mixture formulation. This mixture of linear regressions formulation has appeared in the computer science literature under the name Hierarchical Mixtures of Experts model.This model has been considered from both frequentist and Bayesian viewpoints. We focus on the Bayesian formulation. Previously, estimation of the mixture of linear regression model has been done through straightforward Gibbs sampling with latent variables. This paper contributes to this field in three major areas. First, we provide a theoretical underpinning to the Bayesian implementation by demonstrating consistency of the posterior distribution. This demonstration is done by extending results in Barron, Schervish and Wasserman (Annals of Statistics 27: 536–561, 1999) on bracketing entropy to the regression setting. Second, we demonstrate through examples that straightforward Gibbs sampling may fail to effectively explore the posterior distribution and provide alternative algorithms that are more accurate. Third, we demonstrate the usefulness of the mixture of linear regressions framework in Bayesian robust regression. The methods described in the paper are applied to two examples.     

Fourier smoother and additive models

Suppose the observations (t_i,y_i), i = 1,… n, follow the model where g_j are unknown functions. The estimation of the additive components can be done by approximating g_j, with a function made up of the sum of a linear fit and a truncated Fourier series of cosines and minimizing a penalized least-squares loss function over the coefficients. This finite-dimensional basis approximation, when fitting an additive model with r predictors, has the advantage of reducing the computations drastically, since it does not require the use of the backfitting algorithm. The cross-validation (CV) [or generalized cross-validation (GCV)] for the additive fit is calculated in a further 0(n) operations. A search path in the r-dimensional space of degrees of freedom is proposed along which the CV (GCV) continuously decreases. The path ends when an increase in the degrees of freedom of any of the predictors yields an increase in CV (GCV). This procedure is illustrated on a meteorological data set.     

Mixtures of general location model with factor analyzer covariance structure for clustering mixed type data

Cluster analysis is one of the most widely used method in statistical analyses, in which homogeneous subgroups are identified in a heterogeneous population. Due to the existence of the continuous and discrete mixed data in many applications, so far, some ordinary clustering methods such as, hierarchical methods, k-means and model-based methods have been extended for analysis of mixed data. However, in the available model-based clustering methods, by increasing the number of continuous variables, the number of parameters increases and identifying as well as fitting an appropriate model may be difficult. In this paper, to reduce the number of the parameters, for the model-based clustering mixed data of continuous (normal) and nominal data, a set of parsimonious models is introduced. Models in this set are extended, using the general location model approach, for modeling distribution of mixed variables and applying factor analyzer structure for covariance matrices. The ECM algorithm is used for estimating the parameters of these models. In order to show the performance of the proposed models for clustering, results from some simulation studies and analyzing two real data sets are presented.     

Testing for normality in linear regression models

The importance of the normal distribution for fitting continuous data is well known. However, in many practical situations data distribution departs from normality. For example, the sample skewness and the sample kurtosis are far away from 0 and 3, respectively, which are nice properties of normal distributions. So, it is important to have formal tests of normality against any alternative. D'Agostino et al. [A suggestion for using powerful and informative tests of normality, Am. Statist. 44 (1990), pp. 316–321] review four procedures Z ²(g ₁), Z ²(g ₂), D and K ² for testing departure from normality. The first two of these procedures are tests of normality against departure due to skewness and kurtosis, respectively. The other two tests are omnibus tests. An alternative to the normal distribution is a class of skew-normal distributions (see [A. Azzalini, A class of distributions which includes the normal ones, Scand. J. Statist. 12 (1985), pp. 171–178]). In this paper, we obtain a score test (W) and a likelihood ratio test (LR) of goodness of fit of the normal regression model against the skew-normal family of regression models. It turns out that the score test is based on the sample skewness and is of very simple form. The performance of these six procedures, in terms of size and power, are compared using simulations. The level properties of the three statistics LR, W and Z ²(g ₁) are similar and close to the nominal level for moderate to large sample sizes. Also, their power properties are similar for small departure from normality due to skewness (γ₁≤0.4). Of these, the score test statistic has a very simple form and computationally much simpler than the other two statistics. The LR statistic, in general, has highest power, although it is computationally much complex as it requires estimates of the parameters under the normal model as well as those under the skew-normal model. So, the score test may be used to test for normality against small departure from normality due to skewness. Otherwise, the likelihood ratio statistic LR should be used as it detects general departure from normality (due to both skewness and kurtosis) with, in general, largest power.     

Approximate maximum likelihood estimation in the presence of extra-poisson variation

A two-stage hierarchical model for analysis of discrete data with extra-Poisson variation is examined. The model consists of a Poisson distribution with a mixing lognormal distribution for the mean. A method of approximate maximum likelihood estimation of the parameters is proposed. The method uses the EM algorithm and approximations to facilitate its implementation are derived. Approximate standard errors of the estimates are provided and a numerical example is used to illustrate the method.     

Analysis of the performance of test statistics for detection of outliers (additive,innovative, transient,and level shift) in AR (1) processes

Outlier detection has always been of interest for researchers and data miners. It has been well researched in different knowledge and application domains. This study aims at exploring the correctly identifying outliers using most commonly applied statistics. We evaluate the performance of AO, IO, LS, and TC as vulnerability to spurious outliers by means of empirical level of significance (ELS), power of the test indicating the sensitivity of the statistical tests in detecting changes and the vulnerability to masking of outliers in terms of misspecification frequencies are determined. We have observed that the sampling distribution of test statistic η_tp; tp = AO,?IO,?LS,?TC in case of AR(1) model is connected with the values of n and φ. The sampling distribution of η_TC is less concentrated than the sampling distribution of η_AO, η_IO, and η_LS. In AR(1) process, empirical critical values for 1%, 5%, and 10% upper percentiles are found to be higher than those generally used. We have also found the evidence that the test statistics for transient change (TC) needs to be revisited as the test statistics η_TC is found to be eclipsed by η_AO,?η_LS and η_IO at different δ values. TC keeps on confusing with IO and AO, and at extreme δ values it just gets equal to AO and LS.     

Moments of Order Statistics from Weibull Distribution in the Presence of Multiple Outliers

In this article, we derive exact expressions for the single and product moments of order statistics from Weibull distribution under the contamination model. We assume that X₁, X₂, …, X_{n ? p} are independent with density function f(x) while the remaining, p observations (outliers) X_{n ? p + 1}, …, X_n are independent with density function arises from some modified version of f(x), which is called g(x), in which the location and/or scale parameters have been shifted in value. Next, we investigate the effect of the outliers on the BLUE of the scale parameter. Finally, we deduce some special cases.     

Decomposition of Prediction Error in Multilevel Models

ABSTRACT

We present a decomposition of prediction error for the multilevel model in the context of predicting a future observable y _*j in the jth group of a hierarchical dataset. The multilevel prediction rule is used for prediction and the components of prediction error are estimated via a simulation study that spans the various combinations of level-1 (individual) and level-2 (group) sample sizes and different intraclass correlation values. Additionally, analytical results present the increase in predicted mean square error (PMSE) with respect to prediction error bias. The components of prediction error provide information with respect to the cost of parameter estimation versus data imputation for predicting future values in a hierarchical data set. Specifically, the cost of parameter estimation is very small compared to data imputation.     

Robust Coordinate Descent Algorithm Robust Solution Path for High-dimensional Sparse Regression Modeling

The L₁-type regularization provides a useful tool for variable selection in high-dimensional regression modeling. Various algorithms have been proposed to solve optimization problems for L₁-type regularization. Especially the coordinate descent algorithm has been shown to be effective in sparse regression modeling. Although the algorithm shows a remarkable performance to solve optimization problems for L₁-type regularization, it suffers from outliers, since the procedure is based on the inner product of predictor variables and partial residuals obtained from a non-robust manner. To overcome this drawback, we propose a robust coordinate descent algorithm, especially focusing on the high-dimensional regression modeling based on the principal components space. We show that the proposed robust algorithm converges to the minimum value of its objective function. Monte Carlo experiments and real data analysis are conducted to examine the efficiency of the proposed robust algorithm. We observe that our robust coordinate descent algorithm effectively performs for the high-dimensional regression modeling even in the presence of outliers.     

Density estimation under the koziol-green model of censoring by penalized likelihood methods

We propose two density estimators of the survival distribution in the setting of the Koziol-Green random-censoring model. The estimators are obtained by maximum-penalized-likelihood methods, and we provide an algorithm for their numerical evaluation. We establish the strong consistency of the estimators in the Hellinger metric, the L_p-norms, p= 1,2, ∞, and a Sobolev norm, under mild conditions on the underlying survival density and the censoring distribution.     

Multivariate process capability a bayesian perspective

In this paper an attempt has been made to examine the multivariate versions of the common process capability indices (PCI's) denoted by C_p and C_pk . Markov chain Monte Carlo (MCMC) methods are used to generate sampling distributions for the various PCI's from where inference is performed. Some Bayesian model checking techniques are developed and implemented to examine how well our model fits the data. Finally the methods are exemplified on a historical aircraft data set collected by the Pratt and Whitney Company.     

Robust Bayesian analysis of the binomial empirical Bayes problem

Empirical Bayes estimation is considered for an i.i.d. sequence of binomial parameters θ_i arising from an unknown prior distribution G(.). This problem typically arises in industrial sampling, where samples from lots are routinely used to estimate the lot fraction defective of each lot. Two related issues are explored. The first concerns the fact that only the first few moments of G are typically estimable from the data. This suggests consideration of the interval of estimates (e.g., posterior means) corresponding to the different possible G with the specified moments. Such intervals can be obtained by application of well-known moment theory. The second development concerns the need to acknowledge the uncertainty in the estimation of the first few moments of G. Our proposal is to determine a credible set for the moments, and then find the range of estimates (e.g., posterior means) corresponding to the different possible G with moments in the credible set.     

Projector operators in the multivariate Zyskind-Martin model

Summary Two quadratic formsS _H andS _E for a testable hypothesis and for an error in the multivariate Zyskind-Martin model with singular covariance matrix are expressed by means of projector operators. Thus the results for the multivariate standard model with identity covariance matrix given by Humak (1977) and Christensen (1987, 1991) are generalized for the case of Zyskind-Martin model. Special cases of our results are formulae forS _H andS _E in Aitken's (1935) model. In the case of general Gauss-Markoff modelS _H andS _E can also be expressed by means of projector operators for some subclasses of testable hypotheses. For these hypotheses, testing in Gauss-Markoff model is equivalent to testing in a Zyskind-Martin model.     

Properties of the beta regression model for small area estimation of proportions and application to estimation of poverty rates

Abstract

Linear mixed effects models have been popular in small area estimation problems for modeling survey data when the sample size in one or more areas is too small for reliable inference. However, when the data are restricted to a bounded interval, the linear model may be inappropriate, particularly if the data are near the boundary. Nonlinear sampling models are becoming increasingly popular for small area estimation problems when the normal model is inadequate. This paper studies the use of a beta distribution as an alternative to the normal distribution as a sampling model for survey estimates of proportions which take values in (0, 1). Inference for small area proportions based on the posterior distribution of a beta regression model ensures that point estimates and credible intervals take values in (0, 1). Properties of a hierarchical Bayesian small area model with a beta sampling distribution and logistic link function are presented and compared to those of the linear mixed effect model. Propriety of the posterior distribution using certain noninformative priors is shown, and behavior of the posterior mean as a function of the sampling variance and the model variance is described. An example using 2010 Small Area Income and Poverty Estimates (SAIPE) data is given, and a numerical example studying small sample properties of the model is presented.     

A Novel Estimation Approach for Mixture Transition Distribution Model in High-Order Markov Chains

A transformation is proposed to convert the nonlinear constraints of the parameters in the mixture transition distribution (MTD) model into box-constraints. The proposed transformation removes the difficulties associated with the maximum likelihood estimation (MLE) process in the MTD modeling so that the MLEs of the parameters can be easily obtained via a hybrid algorithm from the evolutionary algorithms and/or quasi-Newton algorithms for global optimization. Simulation studies are conducted to demonstrate MTD modeling by the proposed novel approach through a global search algorithm in R environment. Finally, the proposed approach is used for the MTD modelings of three real data sets.     

Prior Density Selection as a Particular Case of Bayesian Model Selection: A Predictive Approach

A Bayesian model consists of two elements: a sampling model and a prior density. The problem of selecting a prior density is nothing but the problem of selecting a Bayesian model where the sampling model is fixed. A predictive approach is used through a decision problem where the loss function is the squared L ² distance between the sampling density and the posterior predictive density, because the aim of the method is to choose the prior that provides a posterior predictive density as good as possible. An algorithm is developed for solving the problem; this algorithm is based on Lavine's linearization technique.     

A modified adaptive accept–reject algorithm for univariate densities with bounded support

The need to simulate from a univariate density arises in several settings, particularly in Bayesian analysis. An especially efficient algorithm which can be used to sample from a univariate density, f _X , is the adaptive accept–reject algorithm. To implement the adaptive accept–reject algorithm, the user has to envelope T ° f _X , where T is some transformation such that the density g(x) ∝ T ^?1 (α+β x) is easy to sample from. Successfully enveloping T ° f _X , however, requires that the user identify the number and location of T ° f _X ’s inflection points. This is not always a trivial task. In this paper, we propose an adaptive accept–reject algorithm which relieves the user of precisely identifying the location of T ° f _X ’s inflection points. This new algorithm is shown to be efficient and can be used to sample from any density such that its support is bounded and its log is three-times differentiable.     

Simulation algorithms for integrals of a class of sampling distributions arising in population genetics

Efficient stochastic algorithms are presented in order to simulate allele configurations distributed according to a family π _A , 0<A<∞, of exchangeable sampling distributions arising in population genetics. Each distribution π _A has two parameters n and k, the sample size and the number of alleles, respectively. For A→0, the distribution π _A is induced from neutral sampling, whereas for A→∞, it is induced from Maxwell–Boltzmann sampling. Three different Monte Carlo methods (independent sampling procedures) are provided, based on conditioning, sequential methods and a generalization of Pitmans ‘Chinese restaurant process’. Moreover, an efficient Markov chain Monte Carlo method is provided. The algorithms are applied to the homozygosity test and to the Ewens–Watterson–Slatkin test in order to test the hypothesis of selective neutrality.