The Risk of James–Stein and Lasso Shrinkage

This article compares the mean-squared error (or ?₂ risk) of ordinary least squares (OLS), James–Stein, and least absolute shrinkage and selection operator (Lasso) shrinkage estimators in simple linear regression where the number of regressors is smaller than the sample size. We compare and contrast the known risk bounds for these estimators, which shows that neither James–Stein nor Lasso uniformly dominates the other. We investigate the finite sample risk using a simple simulation experiment. We find that the risk of Lasso estimation is particularly sensitive to coefficient parameterization, and for a significant portion of the parameter space Lasso has higher mean-squared error than OLS. This investigation suggests that there are potential pitfalls arising with Lasso estimation, and simulation studies need to be more attentive to careful exploration of the parameter space.     

Bias reduction in the estimation of a shape second-order parameter of a heavy-tailed model

In extreme value theory, the shape second-order parameter is a quite relevant parameter related to the speed of convergence of maximum values, linearly normalized, towards its limit law. The adequate estimation of this parameter is vital for improving the estimation of the extreme value index, the primary parameter in statistics of extremes. In this article, we consider a recent class of semi-parametric estimators of the shape second-order parameter for heavy right-tailed models. These estimators, based on the largest order statistics, depend on a real tuning parameter, which makes them highly flexible and possibly unbiased for several underlying models. In this article, we are interested in the adaptive choice of such tuning parameter and the number of top order statistics used in the estimation procedure. The performance of the methodology for the adaptive choice of parameters is evaluated through a Monte Carlo simulation study.     

A study on tuning parameter selection for the high-dimensional lasso

High-dimensional predictive models, those with more measurements than observations, require regularization to be well defined, perform well empirically, and possess theoretical guarantees. The amount of regularization, often determined by tuning parameters, is integral to achieving good performance. One can choose the tuning parameter in a variety of ways, such as through resampling methods or generalized information criteria. However, the theory supporting many regularized procedures relies on an estimate for the variance parameter, which is complicated in high dimensions. We develop a suite of information criteria for choosing the tuning parameter in lasso regression by leveraging the literature on high-dimensional variance estimation. We derive intuition showing that existing information-theoretic approaches work poorly in this setting. We compare our risk estimators to existing methods with an extensive simulation and derive some theoretical justification. We find that our new estimators perform well across a wide range of simulation conditions and evaluation criteria.     

Model selection and post estimation based on a pretest for logistic regression models

ABSTRACT

This article addresses the problem of parameter estimation of the logistic regression model under subspace information via linear shrinkage, pretest, and shrinkage pretest estimators along with the traditional unrestricted maximum likelihood estimator and restricted estimator. We developed an asymptotic theory for the linear shrinkage and pretest estimators and compared their relative performance using the notion of asymptotic distributional bias and asymptotic quadratic risk. The analytical results demonstrated that the proposed estimation strategies outperformed the classical estimation strategies in a meaningful parameter space. Detailed Monte-Carlo simulation studies were conducted for different combinations and the performance of each estimation method was evaluated in terms of simulated relative efficiency. The results of the simulation study were in strong agreement with the asymptotic analytical findings. Two real-data examples are also given to appraise the performance of the estimators.     

New Ridge Regression Estimator in Semiparametric Regression Models

In the context of ridge regression, the estimation of shrinkage parameter plays an important role in analyzing data. Many efforts have been put to develop the computation of risk function in different full-parametric ridge regression approaches using eigenvalues and then bringing an efficient estimator of shrinkage parameter based on them. In this respect, the estimation of shrinkage parameter is neglected for semiparametric regression model. Not restricted, but the main focus of this approach is to develop necessary tools for computing the risk function of regression coefficient based on the eigenvalues of design matrix in semiparametric regression. For this purpose the differencing methodology is applied. We also propose a new estimator for shrinkage parameter which is of harmonic type mean of ridge estimators. It is shown that this estimator performs better than all the existing ones for the regression coefficient. For our proposal, a Monte Carlo simulation study and a real dataset analysis related to housing attributes are conducted to illustrate the efficiency of shrinkage estimators based on the minimum risk and mean squared error criteria.     

Bayesian Analysis of Stochastic Volatility Models

New techniques for the analysis of stochastic volatility models in which the logarithm of conditional variance follows an autoregressive model are developed. A cyclic Metropolis algorithm is used to construct a Markov-chain simulation tool. Simulations from this Markov chain converge in distribution to draws from the posterior distribution enabling exact finite-sample inference. The exact solution to the filtering/smoothing problem of inferring about the unobserved variance states is a by-product of our Markov-chain method. In addition, multistep-ahead predictive densities can be constructed that reflect both inherent model variability and parameter uncertainty. We illustrate our method by analyzing both daily and weekly data on stock returns and exchange rates. Sampling experiments are conducted to compare the performance of Bayes estimators to method of moments and quasi-maximum likelihood estimators proposed in the literature. In both parameter estimation and filtering, the Bayes estimators outperform these other approaches.     

Model selection and parameter estimation of a multinomial logistic regression model

In the multinomial regression model, we consider the methodology for simultaneous model selection and parameter estimation by using the shrinkage and LASSO (least absolute shrinkage and selection operation) [R. Tibshirani, Regression shrinkage and selection via the LASSO, J. R. Statist. Soc. Ser. B 58 (1996), pp. 267–288] strategies. The shrinkage estimators (SEs) provide significant improvement over their classical counterparts in the case where some of the predictors may or may not be active for the response of interest. The asymptotic properties of the SEs are developed using the notion of asymptotic distributional risk. We then compare the relative performance of the LASSO estimator with two SEs in terms of simulated relative efficiency. A simulation study shows that the shrinkage and LASSO estimators dominate the full model estimator. Further, both SEs perform better than the LASSO estimators when there are many inactive predictors in the model. A real-life data set is used to illustrate the suggested shrinkage and LASSO estimators.     

Shrinkage and penalized estimators in weighted least absolute deviations regression models

In this paper, we consider the estimation problem of the weighted least absolute deviation (WLAD) regression parameter vector when there are some outliers or heavy-tailed errors in the response and the leverage points in the predictors. We propose the pretest and James–Stein shrinkage WLAD estimators when some of the parameters may be subject to certain restrictions. We derive the asymptotic risk of the pretest and shrinkage WLAD estimators and show that if the shrinkage dimension exceeds two, the asymptotic risk of the shrinkage WLAD estimator is strictly less than the unrestricted WLAD estimator. On the other hand, the risk of the pretest WLAD estimator depends on the validity of the restrictions on the parameters. Furthermore, we study the WLAD absolute shrinkage and selection operator (WLAD-LASSO) and compare its relative performance with the pretest and shrinkage WLAD estimators. A simulation study is conducted to evaluate the performance of the proposed estimators relative to that of the unrestricted WLAD estimator. A real-life data example using body fat study is used to illustrate the performance of the suggested estimators.     

Robust adaptive Lasso for variable selection

The adaptive least absolute shrinkage and selection operator (Lasso) and least absolute deviation (LAD)-Lasso are two attractive shrinkage methods for simultaneous variable selection and regression parameter estimation. While the adaptive Lasso is efficient for small magnitude errors, LAD-Lasso is robust against heavy-tailed errors and severe outliers. In this article, we consider a data-driven convex combination of these two modern procedures to produce a robust adaptive Lasso, which not only enjoys the oracle properties, but synthesizes the advantages of the adaptive Lasso and LAD-Lasso. It fully adapts to different error structures including the infinite variance case and automatically chooses the optimal weight to achieve both robustness and high efficiency. Extensive simulation studies demonstrate a good finite sample performance of the robust adaptive Lasso. Two data sets are analyzed to illustrate the practical use of the procedure.     

Meta-analysis,pretest, and shrinkage estimation of kurtosis parameters

An asymptotic theory for the improved estimation of kurtosis parameter vector is developed for multi-sample case using uncertain prior information (UPI) that several kurtosis parameters are the same. Meta-analysis is performed to obtain pooled estimator, as it is a statistical methodology for pooling quantitative evidence. Pooled estimator is a good choice when assumption of homogeneity holds but it becomes inconsistent as assumption violates, therefore pretest and Stein-type shrinkage estimators are proposed as they combine sample and nonsample information in a superior way. Asymptotic properties of suggested estimators are discussed and their risk comparisons are also mentioned.     

Regularized parameter estimation of high dimensional t distribution

We propose penalized-likelihood methods for parameter estimation of high dimensional t distribution. First, we show that a general class of commonly used shrinkage covariance matrix estimators for multivariate normal can be obtained as penalized-likelihood estimator with a penalty that is proportional to the entropy loss between the estimate and an appropriately chosen shrinkage target. Motivated by this fact, we then consider applying this penalty to multivariate t distribution. The penalized estimate can be computed efficiently using EM algorithm for given tuning parameters. It can also be viewed as an empirical Bayes estimator. Taking advantage of its Bayesian interpretation, we propose a variant of the method of moments to effectively elicit the tuning parameters. Simulations and real data analysis demonstrate the competitive performance of the new methods.     

Model Selection for Vector Autoregressive Processes via Adaptive Lasso

Determination of the best subset is an important step in vector autoregressive (VAR) modeling. Traditional methods either conduct subset selection and parameter estimation separately or compute expensively. In this article, we propose a VAR model selection procedure using adaptive Lasso, for it is computational efficient and can select subset and estimate parameters simultaneously. By proper choice of tuning parameters, we can choose the correct subset and obtain the asymptotic normality of the non zero parameters. Simulation studies and real data analysis show that adaptive Lasso performs better than existing methods in VAR model fitting and prediction.     

Shrinkage estimation in replicated median ranked set sampling

In this article, we consider the median ranked set sampling estimation and test of hypothesis for the mean for symmetric distributions. We suggest some alternative estimation strategies for parameters based on shrinkage and pretest principles. It is advantageous to use the non-sample information in the estimation process to construct alternative estimations for the parameter of interest. In this article, large sample properties of the suggested estimators will be assessed numerically using computer simulation. The relative performance of the suggested estimators for moderate and large samples will also be simulated. For illustration purposes, the proposed methodology is applied using data collocated from the Pepsi Cola production company in Al-Khobar, Saudi Arabia.     

Model averaging procedure for varying-coefficient partially linear models with missing responses

This paper is concerned with model averaging procedure for varying-coefficient partially linear models with missing responses. The profile least-squares estimation process and inverse probability weighted method are employed to estimate regression coefficients of the partially restricted models, in which the propensity score is estimated by the covariate balancing propensity score method. The estimators of the linear parameters are shown to be asymptotically normal. Then we develop the focused information criterion, formulate the frequentist model averaging estimators and construct the corresponding confidence intervals. Some simulation studies are conducted to examine the finite sample performance of the proposed methods. We find that the covariate balancing propensity score improves the performance of the inverse probability weighted estimator. We also demonstrate the superiority of the proposed model averaging estimators over those of existing strategies in terms of mean squared error and coverage probability. Finally, our approach is further applied to a real data example.     

On length and area-biased Maxwell distributions

This article addresses the various properties and different methods of estimation of the unknown parameter of length and area-biased Maxwell distributions. Although, our main focus is on estimation from both frequentist and Bayesian point of view, yet, various mathematical and statistical properties of length and area-biased Maxwell distributions (such as moments, moment-generating function (mgf), hazard rate function, mean residual lifetime function, residual lifetime function, reversed residual life function, conditional moments and conditional mgf, stochastic ordering, and measures of uncertainty) are derived. We briefly describe different frequentist approaches, namely, maximum likelihood estimator, moments estimator, least-square and weighted least-square estimators, maximum product of spacings estimator and compare them using extensive numerical simulations. Next we consider Bayes estimation under different types of loss function (symmetric and asymmetric loss functions) using inverted gamma prior for the scale parameter. Furthermore, Bayes estimators and their respective posterior risks are computed and compared using Markov chain Monte Carlo (MCMC) algorithm. Also, bootstrap confidence intervals using frequentist approaches are provided to compare with Bayes credible intervals. Finally, a real dataset has been analyzed for illustrative purposes.     

Estimating and testing effect size from an arbitrary population

In this article, we develop inference tools for an effect size parameter in a paired experiment. A class of estimators is defined that includes natural, shrinkage and shrinkage preliminary test estimators. The shrinkage and preliminary test methods incorporate uncertain prior information on the parameter. This information may be available in the form of a realistic guess on the basis of the experimenter’s knowledge and experience, which can be incorporated into the estimation process to increase the efficiency of the estimator. Asymptotic properties of the proposed estimators are investigated both analytically and computationally. A simulation study is also conducted to assess the performance of the estimators for moderate and large samples. For illustration purposes, the method is applied to a data set.     

Flexible empirical Bayes estimation for wavelets

Wavelet shrinkage estimation is an increasingly popular method for signal denoising and compression. Although Bayes estimators can provide excellent mean-squared error (MSE) properties, the selection of an effective prior is a difficult task. To address this problem, we propose empirical Bayes (EB) prior selection methods for various error distributions including the normal and the heavier-tailed Student t -distributions. Under such EB prior distributions, we obtain threshold shrinkage estimators based on model selection, and multiple-shrinkage estimators based on model averaging. These EB estimators are seen to be computationally competitive with standard classical thresholding methods, and to be robust to outliers in both the data and wavelet domains. Simulated and real examples are used to illustrate the flexibility and improved MSE performance of these methods in a wide variety of settings.     

Simultaneous estimation of Cronbach’s alpha coefficients

The simultaneous estimation of Cronbachs alpha coefficients from q populations under the compound symmetry assumption is considered. In a multi-sample scenario, it is suspected that all the Cronbachs alpha coefficients are identical. Consequently, the inclusion of non-sample information (NSI) on the homogeneity of Cronbachs alpha coefficients in the estimation process may improve precision. We propose improved estimators based on the linear shrinkage, preliminary test, and the Steins type shrinkage strategies, to incorporate available NSI into the estimation. Their asymptotic properties are derived and discussed using the concepts of bias and risk. Extensive Monte-Carlo simulations were conducted to investigate the performance of the estimators.     

Shrinkage estimation in lognormal regression model for censored data

We introduce in this paper, the shrinkage estimation method in the lognormal regression model for censored data involving many predictors, some of which may not have any influence on the response of interest. We develop the asymptotic properties of the shrinkage estimators (SEs) using the notion of asymptotic distributional biases and risks. We show that if the shrinkage dimension exceeds two, the asymptotic risk of the SEs is strictly less than the corresponding classical estimators. Furthermore, we study the penalty (LASSO and adaptive LASSO) estimation methods and compare their relative performance with the SEs. A simulation study for various combinations of the inactive predictors and censoring percentages shows that the SEs perform better than the penalty estimators in certain parts of the parameter space, especially when there are many inactive predictors in the model. It also shows that the shrinkage and penalty estimators outperform the classical estimators. A real-life data example using Worcester heart attack study is used to illustrate the performance of the suggested estimators.     

Small area estimation strategies for large population surveys: a comparison of design and model-based methods

Small area estimation (SAE) concerns with how to reliably estimate population quantities of interest when some areas or domains have very limited samples. This is an important issue in large population surveys, because the geographical areas or groups with only small samples or even no samples are often of interest to researchers and policy-makers. For example, large population health surveys, such as Behavioural Risk Factor Surveillance System and Ohio Mecaid Assessment Survey (OMAS), are regularly conducted for monitoring insurance coverage and healthcare utilization. Classic approaches usually provide accurate estimators at the state level or large geographical region level, but they fail to provide reliable estimators for many rural counties where the samples are sparse. Moreover, a systematic evaluation of the performances of the SAE methods in real-world setting is lacking in the literature. In this paper, we propose a Bayesian hierarchical model with constraints on the parameter space and show that it provides superior estimators for county-level adult uninsured rates in Ohio based on the 2012 OMAS data. Furthermore, we perform extensive simulation studies to compare our methods with a collection of common SAE strategies, including direct estimators, synthetic estimators, composite estimators, and Datta GS, Ghosh M, Steorts R, Maples J.'s [Bayesian benchmarking with applications to small area estimation. Test 2011;20(3):574–588] Bayesian hierarchical model-based estimators. To set a fair basis for comparison, we generate our simulation data with characteristics mimicking the real OMAS data, so that neither model-based nor design-based strategies use the true model specification. The estimators based on our proposed model are shown to outperform other estimators for small areas in both simulation study and real data analysis.