Logistic Regression and Discriminant Analysis by Ordinary Least Squares

If the observations for fitting a polytomous logistic regression model satisfy certain normality assumptions, the maximum likelihood estimates of the regression coefficients are the discriminant function estimates. This article shows that these estimates, their unbiased counterparts, and associated test statistics for variable selection can be calculated using ordinary least squares regression techniques, thereby providing a convenient method for fitting logistic regression models in the normal case. Evidence is given indicating that the discriminant function estimates and test statistics merit wider use in nonnormal cases, especially in exploratory work on large data sets.     

Preliminary phi-divergence test estimators for linear restrictions in a logistic regression model

The problem of estimation of the parameters in a logistic regression model is considered under multicollinearity situation when it is suspected that the parameter of the logistic regression model may be restricted to a subspace. We study the properties of the preliminary test based on the minimum ϕ -divergence estimator as well as in the ϕ -divergence test statistic. The minimum ϕ -divergence estimator is a natural extension of the maximum likelihood estimator and the ϕ -divergence test statistics is a family of the test statistics for testing the hypothesis that the regression coefficients may be restricted to a subspace.     

Retrospective change detection in categorical time series

The aim of this paper is to propose methods of detecting change in the coefficients of a multinomial logistic regression model for categorical time series offline. The alternatives to the null hypothesis of stationarity can be either the hypothesis that it is not true, or that there is a temporary change in the sequence. We use the efficient score vector of the partial likelihood function. This has several advantages. First, the alternative value of the parameter does not have to be estimated; hence, we have a procedure that has a simple structure with only one parameter estimation using all available observations. This is in contrast with the generalized likelihood ratio-based change point tests. The efficient score vector is used in various ways. As a vector, its components correspond to the different components of the multinomial logistic regression model’s parameter vector. Using its quadratic form a test can be defined, where the presence of a change in any or all parameters is tested for. If there are too many parameters one can test for any subset while treating the rest as nuisance parameters. Our motivating example is a DNA sequence of four categories, and our test result shows that in the published data the distribution of the four categories is not stationary.     

Permutation Tests for Linear Models

Several approximate permutation tests have been proposed for tests of partial regression coefficients in a linear model based on sample partial correlations. This paper begins with an explanation and notation for an exact test. It then compares the distributions of the test statistics under the various permutation methods proposed, and shows that the partial correlations under permutation are asymptotically jointly normal with means 0 and variances 1. The method of Freedman & Lane (1983) is found to have asymptotic correlation 1 with the exact test, and the other methods are found to have smaller correlations with this test. Under local alternatives the critical values of all the approximate permutation tests converge to the same constant, so they all have the same asymptotic power. Simulations demonstrate these theoretical results.     

Using entropy distance measures for testing odds ratio parameters under logistic regression models based on case–control data

We propose two retrospective test statistics for testing the vector of odds ratio parameters under the logistic regression model based on case–control data by exploiting the density ratio structure under a two-sample semiparametric model, which is equivalent to the assumed logistic regression model. The proposed test statistics are based on Kullback–Leibler entropy distance and are particularly relevant to the case–control sampling plan. These two test statistics have identical asymptotic chi-squared distributions under the null hypothesis and identical asymptotic noncentral chi-squared distributions under local alternatives to the null hypothesis. Moreover, the proposed test statistics require computation of the maximum semiparametric likelihood estimators of the underlying parameters, but are otherwise easily computed. We present some results on simulation and on the analysis of two real data sets.     

Bayesian adaptive Lasso for quantile regression models with nonignorably missing response data

Abstract

Handling data with the nonignorably missing mechanism is still a challenging problem in statistics. In this paper, we develop a fully Bayesian adaptive Lasso approach for quantile regression models with nonignorably missing response data, where the nonignorable missingness mechanism is specified by a logistic regression model. The proposed method extends the Bayesian Lasso by allowing different penalization parameters for different regression coefficients. Furthermore, a hybrid algorithm that combined the Gibbs sampler and Metropolis-Hastings algorithm is implemented to simulate the parameters from posterior distributions, mainly including regression coefficients, shrinkage coefficients, parameters in the non-ignorable missing models. Finally, some simulation studies and a real example are used to illustrate the proposed methodology.     

Four Correlation Coefficients with a Third Blocking Variable: Their Efficacy,Relative Efficiency,and Test Statistics

Abstract

The efficacy and the asymptotic relative efficiency (ARE) of a weighted sum of Kendall's taus, a weighted sum of Spearman's rhos, a weighted sum of Pearson's r's, and a weighted sum of z-transformation of the Fisher–Yates correlation coefficients, in the presence of a blocking variable, are discussed. The method of selecting the weighting constants that maximize the efficacy of these four correlation coefficients is proposed. The estimate, test statistics and confidence interval of the four correlation coefficients with weights are also developed. To compare the small-sample properties of the four tests, a simulation study is performed. The theoretical and simulated results all prefer the weighted sum of the Pearson correlation coefficients with the optimal weights, as well as the weighted sum of z-transformation of the Fisher–Yates correlation coefficients with the optimal weights.     

Simulation‐based sample‐sizing and power calculations in logistic regression with partial prior information

There have been many approximations developed for sample sizing of a logistic regression model with a single normally‐distributed stimulus. Despite this, it has been recognised that there is no consensus as to the best method. In pharmaceutical drug development, simulation provides a powerful tool to characterise the operating characteristics of complex adaptive designs and is an ideal method for determining the sample size for such a problem. In this paper, we address some issues associated with applying simulation to determine the sample size for a given power in the context of logistic regression. These include efficient methods for evaluating the convolution of a logistic function and a normal density and an efficient heuristic approach to searching for the appropriate sample size. We illustrate our approach with three case studies. Copyright © 2016 John Wiley & Sons, Ltd.     

Goodness-of-fit Tests for GEE with Correlated Binary Data

The marginal logistic regression, in combination with GEE, is an increasingly important method in dealing with correlated binary data. As for independent binary data, when the number of possible combinations of the covariate values in a logistic regression model is much larger than the sample size, such as when the logistic model contains at least one continuous covariate, many existing chi-square goodness-of-fit tests either are not applicable or have some serious drawbacks. In this paper two residual based normal goodness-of-fit test statistics are proposed: the Pearson chi-square and an unweighted sum of residual squares. Easy-to-calculate approximations to the mean and variance of either statistic are also given. Their performance, in terms of both size and power, was satisfactory in our simulation studies. For illustration we apply them to a real data set.     

Testing order restricted hypotheses with proportional hazards

Inferences for survival curves based on right censored continuous or grouped data are studied. Testing homogeneity with an ordered restricted alternative and testing the order restriction as the null hypothesis are considered. Under a proportional hazards model, the ordering on the survival curves corresponds to an ordering on the regression coefficients. Approximate likelihood methods are obtained by applying order restricted procedures to the estimates of the regression coefficients. Ordered analogues to the log rank test which are based on the score statistics are considered also. Chi-bar-squared distributions, which have been studied extensively, are shown to provide reasonable approximations to the null distributions of these tests statistics. Using Monte Carlo techniques, the powers of these two types of tests are compared with those that are available in the literature.     

Ordinary and weighted least-squares estimators

We propose a method of estimating the asymptotic relative efficiency (ARE) of the weighted least-squares estimator (WLSE) with respect to the ordinary least-squares estimator (OLSE) in a heteroscedastic linear regression model with a large number of observations but a small number of replicates at each value of the regressors. The weights used in the WLSE are the reciprocals of the (within-group) average of squared residuals. It is shown that the OLSE is more efficient than the WLSE if the maximum number of replicates is not larger than two. The proposed estimator of the ARE is consistent as the number of observations tends to infinity. Finite-sample performance of this estimator is examined in a simulation study. An adaptive estimator, which is asymptotically more efficient than the OLSE and the WLSE, is proposed.     

Inference in multivariate linear regression models with elliptically distributed errors

In this study we investigate the problem of estimation and testing of hypotheses in multivariate linear regression models when the errors involved are assumed to be non-normally distributed. We consider the class of heavy-tailed distributions for this purpose. Although our method is applicable for any distribution in this class, we take the multivariate t-distribution for illustration. This distribution has applications in many fields of applied research such as Economics, Business, and Finance. For estimation purpose, we use the modified maximum likelihood method in order to get the so-called modified maximum likelihood estimates that are obtained in a closed form. We show that these estimates are substantially more efficient than least-square estimates. They are also found to be robust to reasonable deviations from the assumed distribution and also many data anomalies such as the presence of outliers in the sample, etc. We further provide test statistics for testing the relevant hypothesis regarding the regression coefficients.     

Assessing large sample bias in misspecified model scenarios with reference to exposure model misspecification in errors-in-variable regression: A new computational approach

In this paper, we develop a numerical method for evaluating the large sample bias in estimated regression coefficients arising due to exposure model misspecification while adjusting for measurement errors in errors-in-variable regression. The application of the proposed method has been demonstrated in the case of a logistic errors-in-variable regression model. The method is based on the combination of Monte-Carlo, numerical and, in some special cases, analytic integration techniques. The proposed method facilitates the investigation of the limiting bias in the estimated regression parameters based on a single data set rather than on repeated data sets as required by the conventional repeated sample method. Simulation studies demonstrate that the proposed method provides very similar estimates of bias in the estimated regression parameters under exposure model misspecification in logistic errors-in-variable regression with a higher degree of precision as compared to the conventional repeated sample method.     

CASE-DELETION DIAGNOSTICS FOR TEST STATISTICS IN MULTIVARIATE REGRESSION

Influence measures in multivariate regression analysis have been widely developed, especially through use of the case-deletion approach. However, there seem to be few accounts of the influence of observations on test statistics in hypothesis testing. This paper examines four common multivariate tests, namely the Wilks' ratio, Lawley-Hotelling trace, Pillai's trace and Roy's greatest root for testing a general linear hypothesis of the regression coefficients in multivariate regression. The influence of observations is measured using the case-deletion approach. The proposed diagnostic measures, except that of Roy's greatest root, can be expressed in terms of statistics without involving the actual deletion of observations. An illustrative example is given with satisfactory results.     

Jackknife-After-Bootstrap as Logistic Regression Diagnostic Tool

In this study, we propose using Jackknife-after-Bootstrap (JaB) method to detect influential observations in binary logistic regression model. Performance of the proposed method has been compared with the traditional method for standardized Pearson residuals, Cook's distance, change in the Pearson chi-square and change in the deviance statistics by both real world examples and simulation studies. The results reveal that under the various scenarios considered in this article, JaB performs better than the traditional method and is more robust to masking effect especially for Cook's distance.     

Detection and Estimation of Jump Points in Non parametric Regression Function with AR(1) Noise

In this paper, we propose a method based on wavelet analysis to detect and estimate jump points in non parametric regression function. This method is applied to AR(1) noise process under random design. First, the test statistics are constructed on the empirical wavelet coefficients. Then, under the null hypothesis, the critical values of test statistics are obtained. Under the alternative, the consistency of the test is proved. Afterward, the rate of convergence, the estimators of the number, and locations of change points are given theoretically. Finally, the excellent performance of our method is demonstrated through simulations using artificial and real datasets.     

Cocaine Dependence Treatment Data: Methods for Measurement Error Problems With Predictors Derived From Stationary Stochastic Processes

In a cocaine dependence treatment study, we use linear and nonlinear regression models to model posttreatment cocaine craving scores and first cocaine relapse time. A subset of the covariates are summary statistics derived from baseline daily cocaine use trajectories, such as baseline cocaine use frequency and average daily use amount. These summary statistics are subject to estimation error and can therefore cause biased estimators for the regression coefficients. Unlike classical measurement error problems, the error we encounter here is heteroscedastic with an unknown distribution, and there are no replicates for the error-prone variables or instrumental variables. We propose two robust methods to correct for the bias: a computationally efficient method-of-moments-based method for linear regression models and a subsampling extrapolation method that is generally applicable to both linear and nonlinear regression models. Simulations and an application to the cocaine dependence treatment data are used to illustrate the efficacy of the proposed methods. Asymptotic theory and variance estimation for the proposed subsampling extrapolation method and some additional simulation results are described in the online supplementary material.     

Maximum Likelihood Estimation of Logistic Regression Parameters under Two-phase, Outcome-dependent Sampling

Outcome-dependent sampling increases the efficiency of studies of rare outcomes, examples being case—control studies in epidemiology and choice–based sampling in econometrics. Two-phase or double sampling is a standard technique for drawing efficient stratified samples. We develop maximum likelihood estimation of logistic regression coefficients for a hybrid two-phase, outcome–dependent sampling design. An algorithm is given for determining the estimates by repeated fitting of ordinary logistic regression models. Simulation results demonstrate the efficiency loss associated with alternative pseudolikelihood and weighted likelihood methods for certain data configurations. These results provide an efficient solution to the measurement error problem with validation sampling based on a discrete surrogate.     

Goodness of fit tests for the multiple logistic regression model

Several test statistics are proposed for the purpose of assessing the goodness of fit of the multiple logistic regression model. The test statistics are obtained by applying a chi-square test for a contingency table in which the expected frequencies are determined using two different grouping strategies and two different sets of distributional assumptions. The null distributions of these statistics are examined by applying the theory for chi-square tests of Moore Spruill (1975) and through computer simulations. All statistics are shown to have a chi-square distribution or a distribution which can be well approximated by a chi-square. The degrees of freedom are shown to depend on the particular statistic and the distributional assumptions.

The power of each of the proposed statistics is examined for the normal, linear, and exponential alternative models using computer simulations.