Linearized Ridge Regression Estimator Under the Mean Squared Error Criterion in a Linear Regression Model

This article mainly aims to study the superiority of the notion of linearized ridge regression estimator (LRRE) under the mean squared error criterion in a linear regression model. Firstly, we derive uniform lower bound of MSE for the class of the generalized shrinkage estimator (GSE), based on which it is shown that the optimal LRRE is the best estimator in the class of GSE's. Secondly, we propose the notion of the almost unbiased completeness and show that LRRE possesses such a property. Thirdly, the simulation study is given, from which it indicates that the LRRE performs desirably. Finally, the main results are applied to the well known Hald data.     

Estimating prediction error for complex samples

With a growing interest in using non-representative samples to train prediction models for numerous outcomes it is necessary to account for the sampling design that gives rise to the data in order to assess the generalized predictive utility of a proposed prediction rule. After learning a prediction rule based on a non-uniform sample, it is of interest to estimate the rule's error rate when applied to unobserved members of the population. Efron (1986) proposed a general class of covariance penalty inflated prediction error estimators that assume the available training data are representative of the target population for which the prediction rule is to be applied. We extend Efron's estimator to the complex sample context by incorporating Horvitz–Thompson sampling weights and show that it is consistent for the true generalization error rate when applied to the underlying superpopulation. The resulting Horvitz–Thompson–Efron estimator is equivalent to dAIC, a recent extension of Akaike's information criteria to survey sampling data, but is more widely applicable. The proposed methodology is assessed with simulations and is applied to models predicting renal function obtained from the large-scale National Health and Nutrition Examination Study survey. The Canadian Journal of Statistics 48: 204–221; 2020 © 2019 Statistical Society of Canada     

Penalized and Shrinkage Estimation in the Cox Proportional Hazards Model

This article considers the shrinkage estimation procedure in the Cox's proportional hazards regression model when it is suspected that some of the parameters may be restricted to a subspace. We have developed the statistical properties of the shrinkage estimators including asymptotic distributional biases and risks. The shrinkage estimators have much higher relative efficiency than the classical estimator, furthermore, we consider two penalty estimators—the LASSO and adaptive LASSO—and compare their relative performance with that of the shrinkage estimators numerically. A Monte Carlo simulation experiment is conducted for different combinations of irrelevant predictors and the performance of each estimator is evaluated in terms of simulated mean squared error. Simulation study shows that the shrinkage estimators are comparable to the penalty estimators when the number of irrelevant predictors in the model is relatively large. The shrinkage and penalty methods are applied to two real data sets to illustrate the usefulness of the procedures in practice.     

Hill's estimator for the tail index of an ARMA model

This paper investigates Hill's estimator for the tail index of an ARMA model with i.i.d. residuals. Based on the estimated residuals, it is shown that Hill's estimator is asymptotically normal. This method can achieve a smaller asymptotic variance than applying Hill's estimator to the original data. These results are the same as those in Resnick and Starica (Commun. Statist.—Stochastic Models 13 (4) (1997) 703) for an AR model. However, Resnick and Starica (Commun. Statist.—Stochastic Models 13 (4) (1997) 703) imposed one more condition on the choice of sample fraction than the i.i.d. case. This condition is removed in this paper so that data-driven methods for choosing optimal sample fraction based on i.i.d. data can be applied to our case. As an auxiliary theorem, we establish the weak convergence of the tail empirical process of the estimated residuals, which may be of independent interest.     

Relative performance of stein-rule and preliminary test estimators in linear models least squares theory

In the classical (univariare) linear model, bearing the plausibility of a subset of the regression parameters being close to a pivot, shrinkage least squares estimation of the complementary subset is considered. Based on the usual James-Stein rule, shrinkage least squares estimators are constructed, and under an asymptotic setup (allowing the shrinkage parameters to be 'close to ' the pivot), the relative performance of such estimators and the prcliminary test estimators is studied. In this context, the normality of the errors is also avoided under the same asymptotic setup. None of the shrinkage and preliminary test estimators may dominate the other (in the light of the asymptotic distributional risk criterion, as has been developed here), though each of them fares well relative to the classical least squeres estimator. The chice of the shrinkage factor is also examined properly.     

Developing a Liu estimator for the negative binomial regression model: method and application

This paper introduces a new shrinkage estimator for the negative binomial regression model that is a generalization of the estimator proposed for the linear regression model by Liu [A new class of biased estimate in linear regression, Comm. Stat. Theor. Meth. 22 (1993), pp. 393–402]. This shrinkage estimator is proposed in order to solve the problem of an inflated mean squared error of the classical maximum likelihood (ML) method in the presence of multicollinearity. Furthermore, the paper presents some methods of estimating the shrinkage parameter. By means of Monte Carlo simulations, it is shown that if the Liu estimator is applied with these shrinkage parameters, it always outperforms ML. The benefit of the new estimation method is also illustrated in an empirical application. Finally, based on the results from the simulation study and the empirical application, a recommendation regarding which estimator of the shrinkage parameter that should be used is given.     

Estimating and testing effect size from an arbitrary population

In this article, we develop inference tools for an effect size parameter in a paired experiment. A class of estimators is defined that includes natural, shrinkage and shrinkage preliminary test estimators. The shrinkage and preliminary test methods incorporate uncertain prior information on the parameter. This information may be available in the form of a realistic guess on the basis of the experimenter’s knowledge and experience, which can be incorporated into the estimation process to increase the efficiency of the estimator. Asymptotic properties of the proposed estimators are investigated both analytically and computationally. A simulation study is also conducted to assess the performance of the estimators for moderate and large samples. For illustration purposes, the method is applied to a data set.     

Regularized parameter estimation of high dimensional t distribution

We propose penalized-likelihood methods for parameter estimation of high dimensional t distribution. First, we show that a general class of commonly used shrinkage covariance matrix estimators for multivariate normal can be obtained as penalized-likelihood estimator with a penalty that is proportional to the entropy loss between the estimate and an appropriately chosen shrinkage target. Motivated by this fact, we then consider applying this penalty to multivariate t distribution. The penalized estimate can be computed efficiently using EM algorithm for given tuning parameters. It can also be viewed as an empirical Bayes estimator. Taking advantage of its Bayesian interpretation, we propose a variant of the method of moments to effectively elicit the tuning parameters. Simulations and real data analysis demonstrate the competitive performance of the new methods.     

Efficiency of the QR class estimator in semiparametric regression models to combat multicollinearity

There are some classes of biased estimators for solving the multicollinearity among the predictor variables in statistical literature. In this research, we propose a modified estimator based on the QR decomposition in the semiparametric regression models, to combat the multicollinearity problem of design matrix which makes the data to be less distorted than the other methods. We derive the properties of the proposed estimator, and then, the necessary and sufficient condition for the superiority of the partially generalized QR-based estimator over partially generalized least-squares estimator is obtained. In the biased estimators, selection of shrinkage parameters plays an important role in data analysing. We use generalized cross-validation criterion for selecting the optimal shrinkage parameter and the bandwidth of the kernel smoother. Finally, the Monté-Carlo simulation studies and a real application related to bridge construction data are conducted to support our theoretical discussion.     

Application of shrinkage estimation in linear regression models with autoregressive errors

In this paper, we consider the shrinkage and penalty estimation procedures in the linear regression model with autoregressive errors of order p when it is conjectured that some of the regression parameters are inactive. We develop the statistical properties of the shrinkage estimation method including asymptotic distributional biases and risks. We show that the shrinkage estimators have a significantly higher relative efficiency than the classical estimator. Furthermore, we consider the two penalty estimators: least absolute shrinkage and selection operator (LASSO) and adaptive LASSO estimators, and numerically compare their relative performance with that of the shrinkage estimators. A Monte Carlo simulation experiment is conducted for different combinations of inactive predictors and the performance of each estimator is evaluated in terms of the simulated mean-squared error. This study shows that the shrinkage estimators are comparable to the penalty estimators when the number of inactive predictors in the model is relatively large. The shrinkage and penalty methods are applied to a real data set to illustrate the usefulness of the procedures in practice.     

A lower bound for the risk of classes of shrinkage estimators ina general multivariate estimation problem and some deduced estimators

Given a general statistical model and an arbitrary quadratic loss, we propose a lower bound for the associated risk of a class of shrinkage estimators. With respect to the considered class of shrinkage estimators, this bound is optimal.In the particular case of the estimation of the location parameter of an ellipti-cally symmetric distribution, this bound can be used to find the relative improvement brought by a given estimator and the remaining possible improvement, using a Monte-Carlo method. We deduce from these results a new type of shrinkage estimators whose risk can be as close as one wants of the lower bound near a chosen pole and yet remain bounded. Some of them are good alternatives to the positive-part James-Stein estimator.     

Pooling multivariate data

In case it is doubtful whether two sets of data have the same mean vector, four estimation strategies have been developed for the target mean vector. In this situation, the estimates based on a preliminary test as well as on Stein-rule are advantageous. Two measures of relative efficiency are considered; one based on thequadratic loss function, and the other on the determinant of the mean square error matrix. A max-min rule for the size of the preliminary test of significance is presented. It is demonstrated that the shrinkage estimator dominates the classical estimator, whereas none of the shrinkage estimator and the preliminary test estimator dominate each other. The range in the parameter space where preliminary test estimator dominates shrinkage is investigated analytically and computationally. It is found that the shrinkage estimator outperform the preliminary test estimator except in a region around the null hypothesis. Moreover, for large values of a, the level of statistical significance, shrinkage estimator dominates the preliminary test estimator uniformly. The relative dominance of the estimators is presented.     

Wavelet threshold based on Stein's unbiased risk estimators of restricted location parameter in multivariate normal

In this paper, the problem of estimating the mean vector under non-negative constraints on location vector of the multivariate normal distribution is investigated. The value of the wavelet threshold based on Stein''s unbiased risk estimators is calculated for the shrinkage estimator in restricted parameter space. We suppose that covariance matrix is unknown and we find the dominant class of shrinkage estimators under Balance loss function. The performance evaluation of the proposed class of estimators is checked through a simulation study by using risk and average mean square error values.     

Bayes shrinkage estimator for consistency assessment of treatment effects in multi-regional clinical trials

The primary objective of a multi-regional clinical trial is to investigate the overall efficacy of the drug across regions and evaluate the possibility of applying the overall trial result to some specific region. A challenge arises when there is not enough regional sample size. We focus on the problem of evaluating applicability of a drug to a specific region of interest under the criterion of preserving a certain proportion of the overall treatment effect in the region. We propose a variant of James-Stein shrinkage estimator in the empirical Bayes context for the region-specific treatment effect. The estimator has the features of accommodating the between-region variation and finiteness correction of bias. We also propose a truncated version of the proposed shrinkage estimator to further protect risk in the presence of extreme value of regional treatment effect. Based on the proposed estimator, we provide the consistency assessment criterion and sample size calculation for the region of interest. Simulations are conducted to demonstrate the performance of the proposed estimators in comparison with some existing methods. A hypothetical example is presented to illustrate the application of the proposed method.     

A smoothed Q‐learning algorithm for estimating optimal dynamic treatment regimes

In this paper, we propose a smoothed Q‐learning algorithm for estimating optimal dynamic treatment regimes. In contrast to the Q‐learning algorithm in which nonregular inference is involved, we show that, under assumptions adopted in this paper, the proposed smoothed Q‐learning estimator is asymptotically normally distributed even when the Q‐learning estimator is not and its asymptotic variance can be consistently estimated. As a result, inference based on the smoothed Q‐learning estimator is standard. We derive the optimal smoothing parameter and propose a data‐driven method for estimating it. The finite sample properties of the smoothed Q‐learning estimator are studied and compared with several existing estimators including the Q‐learning estimator via an extensive simulation study. We illustrate the new method by analyzing data from the Clinical Antipsychotic Trials of Intervention Effectiveness–Alzheimer's Disease (CATIE‐AD) study.     

Penalized empirical likelihood inference for sparse additive hazards regression with a diverging number of covariates

High-dimensional sparse modeling with censored survival data is of great practical importance, as exemplified by applications in high-throughput genomic data analysis. In this paper, we propose a class of regularization methods, integrating both the penalized empirical likelihood and pseudoscore approaches, for variable selection and estimation in sparse and high-dimensional additive hazards regression models. When the number of covariates grows with the sample size, we establish asymptotic properties of the resulting estimator and the oracle property of the proposed method. It is shown that the proposed estimator is more efficient than that obtained from the non-concave penalized likelihood approach in the literature. Based on a penalized empirical likelihood ratio statistic, we further develop a nonparametric likelihood approach for testing the linear hypothesis of regression coefficients and constructing confidence regions consequently. Simulation studies are carried out to evaluate the performance of the proposed methodology and also two real data sets are analyzed.     

Adaptive Elastic Net GMM Estimation With Many Invalid Moment Conditions: Simultaneous Model and Moment Selection

This article develops the adaptive elastic net generalized method of moments (GMM) estimator in large-dimensional models with potentially (locally) invalid moment conditions, where both the number of structural parameters and the number of moment conditions may increase with the sample size. The basic idea is to conduct the standard GMM estimation combined with two penalty terms: the adaptively weighted lasso shrinkage and the quadratic regularization. It is a one-step procedure of valid moment condition selection, nonzero structural parameter selection (i.e., model selection), and consistent estimation of the nonzero parameters. The procedure achieves the standard GMM efficiency bound as if we know the valid moment conditions ex ante, for which the quadratic regularization is important. We also study the tuning parameter choice, with which we show that selection consistency still holds without assuming Gaussianity. We apply the new estimation procedure to dynamic panel data models, where both the time and cross-section dimensions are large. The new estimator is robust to possible serial correlations in the regression error terms.     

Stein-type estimation using ranked set sampling

In this study, we consider the application of the James–Stein estimator for population means from a class of arbitrary populations based on ranked set sample (RSS). We consider a basis for optimally combining sample information from several data sources. We succinctly develop the asymptotic theory of simultaneous estimation of several means for differing replications based on the well-defined shrinkage principle. We showcase that a shrinkage-type estimator will have, under quadratic loss, a substantial risk reduction relative to the classical estimator based on simple random sample and RSS. Asymptotic distributional quadratic biases and risks of the shrinkage estimators are derived and compared with those of the classical estimator. A simulation study is used to support the asymptotic result. An over-riding theme of this study is that the shrinkage estimation method provides a powerful extension of its traditional counterpart for non-normal populations. Finally, we will use a real data set to illustrate the computation of the proposed estimators.     

Nonparametric hierarchical Bayes analysis of binomial data via Bernstein polynomial priors

For binomial data analysis, many methods based on empirical Bayes interpretations have been developed, in which a variance‐stabilizing transformation and a normality assumption are usually required. To achieve the greatest model flexibility, we conduct nonparametric Bayesian inference for binomial data and employ a special nonparametric Bayesian prior—the Bernstein–Dirichlet process (BDP)—in the hierarchical Bayes model for the data. The BDP is a special Dirichlet process (DP) mixture based on beta distributions, and the posterior distribution resulting from it has a smooth density defined on [0, 1]. We examine two Markov chain Monte Carlo procedures for simulating from the resulting posterior distribution, and compare their convergence rates and computational efficiency. In contrast to existing results for posterior consistency based on direct observations, the posterior consistency of the BDP, given indirect binomial data, is established. We study shrinkage effects and the robustness of the BDP‐based posterior estimators in comparison with several other empirical and hierarchical Bayes estimators, and we illustrate through examples that the BDP‐based nonparametric Bayesian estimate is more robust to the sample variation and tends to have a smaller estimation error than those based on the DP prior. In certain settings, the new estimator can also beat Stein's estimator, Efron and Morris's limited‐translation estimator, and many other existing empirical Bayes estimators. The Canadian Journal of Statistics 40: 328–344; 2012 © 2012 Statistical Society of Canada