A Diagnostic Test for the Multinomial Logit Model

I propose a Lagrange multiplier test for the multinomial logit model against the dogit model (Gaudry and Dagenais 1979) as the alternative hypothesis. In view of the well-known drawback of the restrictive property of independence from irrelevant alternatives implied by the multinomial logit model, a specification test has much to recommend it. Finite sample properties of the test are studied using a Monte Carlo experiment, and the test's power against the nested multinomial logit model and the multinomial probit model is investigated. The test is found to be sensitive to the values of the regression parameters of the linear random utility function.     

Bayesian robust transformation and variable selection: A unified approach

The authors consider the problem of simultaneous transformation and variable selection for linear regression. They propose a fully Bayesian solution to the problem, which allows averaging over all models considered including transformations of the response and predictors. The authors use the Box‐Cox family of transformations to transform the response and each predictor. To deal with the change of scale induced by the transformations, the authors propose to focus on new quantities rather than the estimated regression coefficients. These quantities, referred to as generalized regression coefficients, have a similar interpretation to the usual regression coefficients on the original scale of the data, but do not depend on the transformations. This allows probabilistic statements about the size of the effect associated with each variable, on the original scale of the data. In addition to variable and transformation selection, there is also uncertainty involved in the identification of outliers in regression. Thus, the authors also propose a more robust model to account for such outliers based on a t‐distribution with unknown degrees of freedom. Parameter estimation is carried out using an efficient Markov chain Monte Carlo algorithm, which permits moves around the space of all possible models. Using three real data sets and a simulated study, the authors show that there is considerable uncertainty about variable selection, choice of transformation, and outlier identification, and that there is advantage in dealing with all three simultaneously. The Canadian Journal of Statistics 37: 361–380; 2009 © 2009 Statistical Society of Canada     

Restricted maximum likelihood estimation of joint mean‐covariance models

The class of joint mean‐covariance models uses the modified Cholesky decomposition of the within subject covariance matrix in order to arrive to an unconstrained, statistically meaningful reparameterisation. The new parameterisation of the covariance matrix has two sets of parameters that separately describe the variances and correlations. Thus, with the mean or regression parameters, these models have three sets of distinct parameters. In order to alleviate the problem of inefficient estimation and downward bias in the variance estimates, inherent in the maximum likelihood estimation procedure, the usual REML estimation procedure adjusts for the degrees of freedom lost due to the estimation of the mean parameters. Because of the parameterisation of the joint mean covariance models, it is possible to adapt the usual REML procedure in order to estimate the variance (correlation) parameters by taking into account the degrees of freedom lost by the estimation of both the mean and correlation (variance) parameters. To this end, here we propose adjustments to the estimation procedures based on the modified and adjusted profile likelihoods. The methods are illustrated by an application to a real data set and simulation studies. The Canadian Journal of Statistics 40: 225–242; 2012 © 2012 Statistical Society of Canada     

Bias prevention of maximum likelihood estimates for scalar skew normal and skew t distributions

The skew normal model is a class of distributions that extends the Gaussian family by including a shape parameter. Despite its nice properties, this model presents some problems with the estimation of the shape parameter. In particular, for moderate sample sizes, the maximum likelihood estimator is infinite with positive probability. As a solution, we use a modified score function as an estimating equation for the shape parameter. It is proved that the resulting modified maximum likelihood estimator is always finite. For confidence intervals a quasi-likelihood approach is considered. When regression and scale parameters are present, the method is combined with maximum likelihood estimators for these parameters. Finally, also the skew t distribution is considered, which may be viewed as an extension of the skew normal. The same method is applied to this model, considering the degrees of freedom as known.     

Inferences from logistic regression models in the presence of small samples,rare events,nonlinearity, and multicollinearity with observational data

The logistic regression model has been widely used in the social and natural sciences and results from studies using this model can have significant policy impacts. Thus, confidence in the reliability of inferences drawn from these models is essential. The robustness of such inferences is dependent on sample size. The purpose of this article is to examine the impact of alternative data sets on the mean estimated bias and efficiency of parameter estimation and inference for the logistic regression model with observational data. A number of simulations are conducted examining the impact of sample size, nonlinear predictors, and multicollinearity on substantive inferences (e.g. odds ratios, marginal effects) when using logistic regression models. Findings suggest that small sample size can negatively affect the quality of parameter estimates and inferences in the presence of rare events, multicollinearity, and nonlinear predictor functions, but marginal effects estimates are relatively more robust to sample size.     

Analytical closed-form solution for binary logit regression by categorical predictors

In contrast to the common belief that the logit model has no analytical presentation, it is possible to find such a solution in the case of categorical predictors. This paper shows that a binary logistic regression by categorical explanatory variables can be constructed in a closed-form solution. No special software and no iterative procedures of nonlinear estimation are needed to obtain a model with all its parameters and characteristics, including coefficients of regression, their standard errors and t-statistics, as well as the residual and null deviances. The derivation is performed for logistic models with one binary or categorical predictor, and several binary or categorical predictors. The analytical formulae can be used for arithmetical calculation of all the parameters of the logit regression. The explicit expressions for the characteristics of logit regression are convenient for the analysis and interpretation of the results of logistic modeling.     

Model-Robust Design of Conjoint Choice Experiments

Within the context of choice experimental designs, most authors have proposed designs for the multinomial logit model under the assumption that only the main effects matter. Very little attention has been paid to designs for attribute interaction models. In this article, three types of Bayesian D-optimal designs for the multinomial logit model are studied: main-effects designs, interaction-effects designs, and composite designs. Simulation studies are used to show that in situations where a researcher is not sure whether or not attribute interaction effects are present, it is best to take into account interactions in the design stage. In particular, it is shown that a composite design constructed by including an interaction-effects model and a main-effects model in the design criterion is most robust against misspecification of the underlying model when it comes to making precise predictions.     

Fixed Effects and Bias Due to the Incidental Parameters Problem in the Tobit Model

The maximum likelihood estimator (MLE) in nonlinear panel data models with fixed effects is widely understood (with a few exceptions) to be biased and inconsistent when T, the length of the panel, is small and fixed. However, there is surprisingly little theoretical or empirical evidence on the behavior of the estimator on which to base this conclusion. The received studies have focused almost exclusively on coefficient estimation in two binary choice models, the probit and logit models. In this note, we use Monte Carlo methods to examine the behavior of the MLE of the fixed effects tobit model. We find that the estimator's behavior is quite unlike that of the estimators of the binary choice models. Among our findings are that the location coefficients in the tobit model, unlike those in the probit and logit models, are unaffected by the “incidental parameters problem.” But, a surprising result related to the disturbance variance emerges instead - the finite sample bias appears here rather than in the slopes. This has implications for estimation of marginal effects and asymptotic standard errors, which are also examined in this paper. The effects are also examined for the probit and truncated regression models, extending the range of received results in the first of these beyond the widely cited biases in the coefficient estimators.     

Comparison of Separable Components in Different Samples

Abstract.  Imagine we have two different samples and are interested in doing semi- or non-parametric regression analysis in each of them, possibly on the same model. In this paper, we consider the problem of testing whether a specific covariate has different impacts on the regression curve in these two samples. We compare the regression curves of different samples but are interested in specific differences instead of testing for equality of the whole regression function. Our procedure does allow for random designs, different sample sizes, different variance functions, different sets of regressors with different impact functions, etc. As we use the marginal integration approach, this method can be applied to any strong, weak or latent separable model as well as to additive interaction models to compare the lower dimensional separable components between the different samples. Thus, in the case of having separable models, our procedure includes the possibility of comparing the whole regression curves, thereby avoiding the curse of dimensionality. It is shown that bootstrap fails in theory and practice. Therefore, we propose a subsampling procedure with automatic choice of subsample size. We present a complete asymptotic theory and an extensive simulation study.     

Geometry and Degrees of Freedom of Linearly Constrained Generalized Lasso

The least squares fit in a linear regression is always unique even when the design matrix has rank deficiency. In this paper, we extend this classic result to linearly constrained generalized lasso. It is shown that under a mild condition, the fit can be represented as a projection onto a polytope and, hence, is unique no matter whether design matrix X has full column rank or not. Furthermore, a formula for the degrees of freedom is derived to characterize the effective number of parameters. It directly yields an unbiased estimate of degrees of freedom, which can be incorporated in an information criterion for model selection.     

The Effect of Estimation of Unknown Degrees of Freedom on Estimation of Remaining Parameters in Spanos’ Conditional t Heteroskedastic Model

This article investigates the effect of estimation of unknown degrees of freedom on efficient estimation of remaining parameters in Spanos’ conditional t heteroskedastic model. We compare by simulation three maximum likelihood estimators (MLEs) of the remaining parameters in the model: the MLE of the remaining parameters when all the parameters are estimated by the MLE, when the degrees of freedom is estimated by a method of moments estimator, and when the degrees of freedom is erroneously specified. The latter two methods are found to perform poorly compared to the former method for the inference of variance parameters in the model. Thus, efficient estimation of degrees of freedom by the MLE is important to estimate efficiently the remaining variance parameters.     

Optimal designs for beta-binomial logistic regression models

Optimal designs for a logistic regression model with over-dispersion introduced by a beta-binomial distribution are characterized. Designs are defined by a set of design points and design weights as usual but, in addition, the experimenter must also make a choice of a sub-sampling design specifying the distribution of observations on sample sizes. In an earlier work it has been shown that Ds-optimal sampling designs for estimation of the parameters of the beta-binomial distribution are supported on at most two design points. This admits a simplified approach using single sample sizes. Linear predictor values for Ds-optimal designs using a common sample size are tabulated for different levels of over-dispersion and choice of subsets of parameters.     

Estimation of the parameters of a regression model with a multivariate t error variable

This paper deals with a regression model for several vari¬ables under the assumption that the errors have a multivariate t-distribution. The parameters of the model, the regression parameters, as well as the scale parameters and the degress of freedom of the error variable are estimated and the estimation procedure is illustrated by a numerical example, Also, the prop¬erties of the estimators and tests for the regression parameters are discussed.     

Tuning Parameter Selection in Cox Proportional Hazards Model with a Diverging Number of Parameters

Regularized variable selection is a powerful tool for identifying the true regression model from a large number of candidates by applying penalties to the objective functions. The penalty functions typically involve a tuning parameter that controls the complexity of the selected model. The ability of the regularized variable selection methods to identify the true model critically depends on the correct choice of the tuning parameter. In this study, we develop a consistent tuning parameter selection method for regularized Cox's proportional hazards model with a diverging number of parameters. The tuning parameter is selected by minimizing the generalized information criterion. We prove that, for any penalty that possesses the oracle property, the proposed tuning parameter selection method identifies the true model with probability approaching one as sample size increases. Its finite sample performance is evaluated by simulations. Its practical use is demonstrated in The Cancer Genome Atlas breast cancer data.     

A Mixed Model-assisted Regression Estimator that Uses Variables Employed at the Design Stage

The Generalized regression estimator (GREG) of a finite population mean or total has been shown to be asymptotically optimal when the working linear regression model upon which it is based includes variables related to the sampling design. In this paper a regression estimator assisted by a linear mixed superpopulation model is proposed. It accounts for the extra information coming from the design in the random component of the model and saves degrees of freedom in finite sample estimation. This procedure combines the larger asymptotic efficiency of the optimal estimator and the greater finite sample stability of the GREG. Design based properties of the proposed estimator are discussed and a small simulation study is conducted to explore its finite sample performance.     

Prediction of response values in linear regression models from replicated experiments

This paper considers the problem of prediction in a linear regression model when data sets are available from replicated experiments. Pooling the data sets for the estimation of regression parameters, we present three predictors — one arising from the least squares method and two stemming from Stein-rule method. Efficiency properties of these predictors are discussed when they are used to predict actual and average values of response variable within/outside the sample. Received: November 17, 1999; revised version: August 10, 2000     

F-tests with a rare pathology

ANOVA is routinely used to compare pathologies. Nevertheless, in many situations, the sample dimensions may not be known when planning the study. This is specially relevant when one of the pathologies is rare. Thus, the sample size for that pathology or for all pathologies must be considered as random. Sample selection for the non-rare pathologies may be carried out to increase the balance of the model. This leads to F-tests with random non-centrality parameters and random degrees of freedom for the errors. The distribution of such tests statistics is obtained.     

Multi - sample test of equal gamma distribution scale parameters in presence of unknown common shape parameter

Shiue and Bain proposed an approximate F statistic for testing equality of two gamma distribution scale parameters in presence of a common and unknown shape parameter. By generalizing Shiue and Bain's statistic we develop a new statistic for testing equality of L >= 2 gamma distribution scale parameters. We derive the distribution of the new statistic ESP for L = 2 and equal sample size situation. For other situations distribution of ESP is not known and test based on the ESP statistic has to be performed by using simulated critical values. We also derive a C(α) statistic CML and develop a likelihood ratio statistic, LR, two modified likelihood ratio statistics M and MLB and a quadratic statistic Q. The distribution of each of the statistics CML, LR, M, MLB and Q is asymptotically chi-square with L - 1 degrees of freedom. We then conducted a monte-carlo simulation study to compare the perfor- mance of the statistics ESP, LR, M, MLB, CML and Q in terms of size and power. The statistics LR, M, MLB and Q are in general liberal and do not show power advantage over other statistics. The statistic CML, based on its asymptotic chi-square distribution, in general, holds nominal level well. It is most powerful or nearly most powerful in most situations and is simple to use. Hence, we recommend the statistic CML for use in general. For better power the statistic ESP, based on its empirical distribution, is recommended for the special situation for which there is evidence in the data that λ₁ < … < λ_L and n₁ < … < n_L, where λ₁ …, λ_L are the scale parameters and n₁,…, n_L are the sample sizes.     

Estimation of a scale parameter in mixture models with unknown location

Estimation of the scale parameter in mixture models with unknown location is considered under Stein's loss. Under certain conditions, the inadmissibility of the “usual” estimator is established by exhibiting better estimators. In addition, robust improvements are found for a specified submodel of the original model. The results are applied to mixtures of normal distributions and mixtures of exponential distributions. Improved estimators of the variance of a normal distribution are shown to be robust under any scale mixture of normals having variance greater than the variance of that normal distribution. In particular, Stein's (Ann. Inst. Statist. Math. 16 (1964) 155) and Brewster's and Zidek's (Ann. Statist. 2 (1974) 21) estimators obtained under the normal model are robust under the t model, for arbitrary degrees of freedom, and under the double-exponential model. Improved estimators for the variance of a t distribution with unknown and arbitrary degrees of freedom are also given. In addition, improved estimators for the scale parameter of the multivariate Lomax distribution (which arises as a certain mixture of exponential distributions) are derived and the robustness of Zidek's (Ann. Statist. 1 (1973) 264) and Brewster's (Ann. Statist. 2 (1974) 553) estimators of the scale parameter of an exponential distribution is established under a class of modified Lomax distributions.     

Robustness of the student t based M-estimator

This paper considers the maximum likelihood type (M) estimator based on Student's t distribution for the location/scale model. The Student t M-estimator is generally thought to be robust to outliers. This paper shows that this is only true if the degrees of freedom parameter is kept fixed. By contrast, if the degrees of freedom parameter is also estimated from the data, the influence functions for the scale and degrees of freedom parameter become unbounded. Moreover, the influence function of the location parameter remains bounded, but its change-of-variance function is unboi~nded. The intuitioil behind these results is explained in the paper. The rates at which both the influence functions and the change-of-variance function diverge to infinity, are very slow. Tliis implies that outliers have to be extremely large in order to become detrimental to the performance of the Student t based M-estimator with estimated degrees of freedom. The theoretical results are illustrated in a a simulation experiment using several related competing estimators and several distributions for the error process.