On Variational Bayes Estimation and Variational Information Criteria for Linear Regression Models

Variational Bayes (VB) estimation is a fast alternative to Markov Chain Monte Carlo for performing approximate Baesian inference. This procedure can be an efficient and effective means of analyzing large datasets. However, VB estimation is often criticised, typically on empirical grounds, for being unable to produce valid statistical inferences. In this article we refute this criticism for one of the simplest models where Bayesian inference is not analytically tractable, that is, the Bayesian linear model (for a particular choice of priors). We prove that under mild regularity conditions, VB based estimators enjoy some desirable frequentist properties such as consistency and can be used to obtain asymptotically valid standard errors. In addition to these results we introduce two VB information criteria: the variational Akaike information criterion and the variational Bayesian information criterion. We show that variational Akaike information criterion is asymptotically equivalent to the frequentist Akaike information criterion and that the variational Bayesian information criterion is first order equivalent to the Bayesian information criterion in linear regression. These results motivate the potential use of the variational information criteria for more complex models. We support our theoretical results with numerical examples.     

THE SMALL SAMPLE PROPERTIES OF THE PRINCIPAL COMPONENTS ESTIMATOR FOR REGRESSION COEFFICIENTS

ABSTRACT

Under the mean square error matrix (MSEM) criterion and Pitman closeness (PC) criterion, the principal components estimator (PRCE) is compared with the least square estimator (LSE) and the superiority of PRCE over LSE is achieved respectively. Finally, we examine the superiority of PRCE predictor over the LSE predictor based on these two criteria.     

The Superiorities of Bayes Linear Minimum Risk Estimation in Linear Model

In this article, the Bayes linear minimum risk estimator (BLMRE) of parameters is derived in linear model. The superiorities of the BLMRE over ordinary least square estimator (LSE) is studied in terms of the mean square error matrix (MSEM) criterion and Pitman closeness (PC) criterion.     

Improved Liu estimator in a linear regression model

In this paper, we mainly aim to introduce the notion of improved Liu estimator (ILE) in the linear regression model y=Xβ+e. The selection of the biasing parameters is investigated under the PRESS criterion and the optimal selection is successfully derived. We make a simulation study to show the performance of ILE compared to the ordinary least squares estimator and the Liu estimator. Finally, the main results are applied to the Hald data.     

Nonparametric approach to intervention time series modeling

Time series are often affected by interventions such as strikes, earthquakes, or policy changes. In the current paper, we build a practical nonparametric intervention model using the central mean subspace in time series. We estimate the central mean subspace for time series taking into account known interventions by using the Nadaraya–Watson kernel estimator. We use the modified Bayesian information criterion to estimate the unknown lag and dimension. Finally, we demonstrate that this nonparametric approach for intervened time series performs well in simulations and in a real data analysis such as the Monthly average of the oxidant.     

On the weighted mixed Liu-type estimator under unbiased stochastic restrictions

In this article, we introduce the weighted mixed Liu-type estimator (WMLTE) based on the weighted mixed and Liu-type estimator (LTE) in linear regression model. We will also present necessary and sufficient conditions for superiority of the weighted mixed Liu-type estimator over the weighted mixed estimator (WME) and Liu type estimator (LTE) in terms of mean square error matrix (MSEM) criterion. Finally, a numerical example and a Monte Carlo simulation is also given to show the theoretical results.     

Performance of the principal component two-parameter estimator in misspecified linear regression model

In this article, we consider the performance of the principal component two-parameter estimator in situation of multicollinearity for misspecified linear regression model where misspecification is due to omission of some relevant explanatory variables. The conditions of superiority of the principal component two-parameter estimator over some estimators under the Mahalanobis loss function by the average loss criterion are derived. Furthermore, a real data example and a Monte Carlo simulation study are provided to illustrate some of the theoretical results.     

Estimation in a linear regression model with stochastic linear restrictions: a new two-parameter-weighted mixed estimator

The present paper considers the weighted mixed regression estimation of the coefficient vector in a linear regression model with stochastic linear restrictions binding the regression coefficients. We introduce a new two-parameter-weighted mixed estimator (TPWME) by unifying the weighted mixed estimator of Schaffrin and Toutenburg [1] and the two-parameter estimator (TPE) of Özkale and Kaç?ranlar [2]. This new estimator is a general estimator which includes the weighted mixed estimator, the TPE and the restricted two-parameter estimator (RTPE) proposed by Özkale and Kaç?ranlar [2] as special cases. Furthermore, we compare the TPWME with the weighted mixed estimator and the TPE with respect to the matrix mean square error criterion. A numerical example and a Monte Carlo simulation experiment are presented by using different estimators of the biasing parameters to illustrate some of the theoretical results.     

Adaptive-Cox model averaging for right-censored data

In medical studies, Cox proportional hazards model is a commonly used method to deal with the right-censored survival data accompanied by many explanatory covariates. In practice, the Akaike's information criterion (AIC) or the Bayesian information criterion (BIC) is usually used to select an appropriate subset of covariates. It is well known that neither the AIC criterion nor the BIC criterion dominates for all situations. In this paper, we propose an adaptive-Cox model averaging procedure to get a more robust hazard estimator. First, by applying AIC and BIC criteria to perturbed datasets, we obtain two model averaging (MA) estimated survival curves, called AIC-MA and BIC-MA. Then, based on Kullback–Leibler loss, a better estimate of survival curve between AIC-MA and BIC-MA is chosen, which results in an adaptive-Cox estimate of survival curve. Simulation results show the superiority of our approach and an application of the proposed method is also presented by analyzing the German Breast Cancer Study dataset.     

Forecasting time series of economic processes by model averaging across data frames of various lengths

This paper presents an extension of mean-squared forecast error (MSFE) model averaging for integrating linear regression models computed on data frames of various lengths. Proposed method is considered to be a preferable alternative to best model selection by various efficiency criteria such as Bayesian information criterion (BIC), Akaike information criterion (AIC), F-statistics and mean-squared error (MSE) as well as to Bayesian model averaging (BMA) and naïve simple forecast average. The method is developed to deal with possibly non-nested models having different number of observations and selects forecast weights by minimizing the unbiased estimator of MSFE. Proposed method also yields forecast confidence intervals with a given significance level what is not possible when applying other model averaging methods. In addition, out-of-sample simulation and empirical testing proves efficiency of such kind of averaging when forecasting economic processes.     

Adjustive Liu-Type Estimators in Linear Regression Models

In this article, we aim to put forward the notion of adjustive Liu-type estimator (ALTE) in the linear regression model. First, the explicit expression of the optimal selection of the adjustive factors is derived under the PRESS criterion through matrix techniques. Then, the results are applied to the dataset on Portland cement. Moreover, to select biasing parameters from the theoretical point of view, we extend ALTE to the generalized version (GALTE) and obtained the optimal ones. The results of the Portland cement data show that ALTE's and GALTE's can substantially improve the ordinary least squares estimator and Liu-type estimators.     

Restricted ridge estimator in the logistic regression model

It is known that when the multicollinearity exists in the logistic regression model, variance of maximum likelihood estimator is unstable. As a remedy, Schaefer et al. presented a ridge estimator in the logistic regression model. Making use of the ridge estimator, when some linear restrictions are also present, we introduce a restricted ridge estimator in the logistic regression model. Statistical properties of this newly defined estimator will be studied and comparisons are done in the simulation study in the sense of mean squared error criterion. A real-data example and a simulation study are introduced to discuss the performance of this estimator.     

Performance of Variable Selection Methods in Regression Using Variations of the Bayesian Information Criterion

The Bayesian information criterion (BIC) is widely used for variable selection. We focus on the regression setting for which variations of the BIC have been proposed. A version that includes the Fisher Information matrix of the predictor variables performed best in one published study. In this article, we extend the evaluation, introduce a performance measure involving how closely posterior probabilities are approximated, and conclude that the version that includes the Fisher Information often favors regression models having more predictors, depending on the scale and correlation structure of the predictor matrix. In the image analysis application that we describe, we therefore prefer the standard BIC approximation because of its relative simplicity and competitive performance at approximating the true posterior probabilities.     

Consistent Bayesian information criterion based on a mixture prior for possibly high-dimensional multivariate linear regression models

In the problem of selecting variables in a multivariate linear regression model, we derive new Bayesian information criteria based on a prior mixing a smooth distribution and a delta distribution. Each of them can be interpreted as a fusion of the Akaike information criterion (AIC) and the Bayesian information criterion (BIC). Inheriting their asymptotic properties, our information criteria are consistent in variable selection in both the large-sample and the high-dimensional asymptotic frameworks. In numerical simulations, variable selection methods based on our information criteria choose the true set of variables with high probability in most cases.     

Comparisons of the <Emphasis Type="Italic">r</Emphasis> − <Emphasis Type="Italic">k</Emphasis> class estimator to the ordinary least squares estimator under the Pitman’s closeness criterion

In the presence of multicollinearity, the r − k class estimator is proposed as an alternative to the ordinary least squares (OLS) estimator which is a general estimator including the ordinary ridge regression (ORR), the principal components regression (PCR) and the OLS estimators. Comparison of competing estimators of a parameter in the sense of mean square error (MSE) criterion is of central interest. An alternative criterion to the MSE criterion is the Pitman’s (1937) closeness (PC) criterion. In this paper, we compare the r − k class estimator to the OLS estimator in terms of PC criterion so that we can get the comparison of the ORR estimator to the OLS estimator under the PC criterion which was done by Mason et al. (1990) and also the comparison of the PCR estimator to the OLS estimator by means of the PC criterion which was done by Lin and Wei (2002).     

Comparison of Two Estimators of Parameters Under Pitman Nearness Criterion

In this article, Pitman nearness criterion is used to compare two competing united biased estimators in linear model. In particular, a sufficient and necessary condition for one estimator being superior to the other is derived. Furthermore, a simulation study is performed to illustrate the theoretical results and several special cases are also studied.     

Information criteria for inhomogeneous spatial point processes

The theoretical foundation for a number of model selection criteria is established in the context of inhomogeneous point processes and under various asymptotic settings: infill, increasing domain and combinations of these. For inhomogeneous Poisson processes we consider Akaike's information criterion and the Bayesian information criterion, and in particular we identify the point process analogue of ‘sample size’ needed for the Bayesian information criterion. Considering general inhomogeneous point processes we derive new composite likelihood and composite Bayesian information criteria for selecting a regression model for the intensity function. The proposed model selection criteria are evaluated using simulations of Poisson processes and cluster point processes.     

Bayesian D-optimal design for logistic regression model with exponential distribution for random intercept

ABSTRACT

Nowadays, generalized linear models have many applications. Some of these models which have more applications in the real world are the models with random effects; that is, some of the unknown parameters are considered random variables. In this article, this situation is considered in logistic regression models with a random intercept having exponential distribution. The aim is to obtain the Bayesian D-optimal design; thus, the method is to maximize the Bayesian D-optimal criterion. For the model was considered here, this criterion is a function of the quasi-information matrix that depends on the unknown parameters of the model. In the Bayesian D-optimal criterion, the expectation is acquired in respect of the prior distributions that are considered for the unknown parameters. Thus, it will only be a function of experimental settings (support points) and their weights. The prior distribution of the fixed parameters is considered uniform and normal. The Bayesian D-optimal design is finally calculated numerically by R3.1.1 software.     

Mean Squared Error Matrix Comparisons of Some Restricted Almost Unbiased Estimators

In this article, we introduce two almost unbiased estimators for the vector of unknown parameters in a linear regression model when additional linear restrictions on the parameter vector are assumed to hold. Superiority of the two estimators under the mean squared error matrix (MSEM) is discussed. Furthermore, a numerical example and simulation study are given to illustrate some of the theoretical results.