Selection of Variables in Multivariate Regression Models for Large Dimensions

The Akaike information criterion, AIC, and Mallows’ C _p statistic have been proposed for selecting a smaller number of regressors in the multivariate regression models with fully unknown covariance matrix. All of these criteria are, however, based on the implicit assumption that the sample size is substantially larger than the dimension of the covariance matrix. To obtain a stable estimator of the covariance matrix, it is required that the dimension of the covariance matrix is much smaller than the sample size. When the dimension is close to the sample size, it is necessary to use ridge-type estimators for the covariance matrix. In this article, we use a ridge-type estimators for the covariance matrix and obtain the modified AIC and modified C _p statistic under the asymptotic theory that both the sample size and the dimension go to infinity. It is numerically shown that these modified procedures perform very well in the sense of selecting the true model in large dimensional cases.     

Post model-fitting exploration via a “Next-Door” analysis

We propose a simple method for evaluating the model that has been chosen by an adaptive regression procedure, our main focus being the lasso. This procedure deletes each chosen predictor and refits the lasso to get a set of models that are “close” to the chosen “base model,” and compares the error rates of the base model with that of nearby models. If the deletion of a predictor leads to significant deterioration in the model's predictive power, the predictor is called indispensable; otherwise, the nearby model is called acceptable and can serve as a good alternative to the base model. This provides both an assessment of the predictive contribution of each variable and a set of alternative models that may be used in place of the chosen model. We call this procedure “Next-Door analysis” since it examines models “next” to the base model. It can be applied to supervised learning problems with ℓ₁ penalization and stepwise procedures. We have implemented it in the R language as a library to accompany the well-known glmnet library. The Canadian Journal of Statistics 48: 447–470; 2020 © 2020 Statistical Society of Canada     

THE OPTIMAL TEST FOR SELECTING THE GREATER OF TWO BINOMIAL PROBABILITIES

The efficiency of a sequential test is related to the “importance” of the trials within the test. This relationship is used to find the optimal test for selecting the greater of two binomial probabilities, p_α and p_b, namely, the stopping rule is “gambler's ruin” and the optimal discipline when p_α+p_b≤ 1 (≥ 1) is play-the-winner (loser), i.e. an α-trial which results in a success is followed by an α-trial (b-trial) whereas an α-trial which results in a failure is followed by α b-trid (α-trial) and correspondingly for b-trials.     

A note on L1 consistent estimation

Let (??, ??) be a space with a σ-field, M = {P_s; s o} a family of probability measures on A, Θ arbitrary, X₁,…,X_n independently and identically distributed P random variables. Metrize Θ with the L₁ distance between measures, and assume identifiability. Minimum-distance estimators are constructed that relate rates of convergence with Vapnik-Cervonenkis exponents when M is “regular”. An alternative construction of estimates is offered via Kolmogorov's chain argument.     

Asymptotic efficiency of model selection criteria: the nonzero mean gaussian ar(∞) case

Motivated by Shibata’s (1980) asymptotic efficiency results this paper dis-cusses the asymptotic efficiency of the order selected by a selection procedure for an infinite order autoregressive process with nonzero mean and unob servable errors that constitute a sequence of independent Gaussian random variables with mean zero and variance σ² The asymptotic efficiency is established for AIC–type selection criteria such as AIC’, FPE, and S_n(k). In addition, some asymptotic results about the estimators of the parameters of the process and the error–sequence are presented.     

Measuring the complexity of generalized linear hierarchical models

Measuring a statistical model's complexity is important for model criticism and comparison. However, it is unclear how to do this for hierarchical models due to uncertainty about how to count the random effects. The authors develop a complexity measure for generalized linear hierarchical models based on linear model theory. They demonstrate the new measure for binomial and Poisson observables modeled using various hierarchical structures, including a longitudinal model and an areal‐data model having both spatial clustering and pure heterogeneity random effects. They compare their new measure to a Bayesian index of model complexity, the effective number p_D of parameters (Spiegelhalter, Best, Carlin & van der Linde 2002); the comparisons are made in the binomial and Poisson cases via simulation and two real data examples. The two measures are usually close, but differ markedly in some instances where p_D is arguably inappropriate. Finally, the authors show how the new measure can be used to approach the difficult task of specifying prior distributions for variance components, and in the process cast further doubt on the commonly‐used vague inverse gamma prior.     

Generalized correlation and kernel causality with applications in development economics

New generalized correlation measures of 2012, GMC(Y|X), use Kernel regressions to overcome the linearity of Pearson's correlation coefficients. A new matrix of generalized correlation coefficients is such that when |r*_ij| > |r*_ji|, it is more likely that the column variable X_j is what Granger called the “instantaneous cause” or what we call “kernel cause” of the row variable X_i. New partial correlations ameliorate confounding. Various examples and simulations support robustness of new causality. We include bootstrap inference, robustness checks based on the dependence between regressor and error, and on the out-of-sample forecasts. Data for 198 countries on nine development variables support growth policy over redistribution and Deaton's criticism of foreign aid. Potential applications include Big Data, since our R code is available in the online supplementary material.     

Set estimation and nonparametric detection

The authors consider the estimation of a set S ? R^d from a random sample of n points. They examine the properties of a detection method, proposed by Devroye & Wise (1980), which relies on the use of a “naive” estimator of S defined as a union of balls centered at the sample points with common radius ?_n. They obtain the convergence rate for the probability of false alarm and show that the smoothing parameter ?_n can be used to incorporate some prior information on the shape of S. They suggest two general methods for selecting ?_n and illustrate them with a simulation study and a real data example.     

Changepoint Problems Under Random Censorship

Let X ₁, …, X _[nθ], X _{[nθ] + 1}, …, X _n be a sequence of independent random variables (the “lifetimes”) such that X _j ? F ₁ for 1 ≤ j ≤ [nθ] and X _j ? F ₂ for [nθ] + 1 ≤ j ≤ n, with F ₁ ≠ F ₂ unknown. In this paper we investigate an estimator θ _n for the changepoint θ if the X's are subject to censoring. The rate of almost sure convergence of θ _n to θ is established and a test for the hypothesis θ = 0, i.e. “no change”, is proposed.     

Confidence Intervals for Nonparametric Regression Functions with Missing Data

Suppose that we have a nonparametric regression model Y = m(X) + ε with X ∈ R^p, where X is a random design variable and is observed completely, and Y is the response variable and some Y-values are missing at random. Based on the “complete” data sets for Y after nonaprametric regression imputation and inverse probability weighted imputation, two estimators of the regression function m(x₀) for fixed x₀ ∈ R^p are proposed. Asymptotic normality of two estimators is established, which is used to construct normal approximation-based confidence intervals for m(x₀). We also construct an empirical likelihood (EL) statistic for m(x₀) with limiting distribution of χ²₁, which is used to construct an EL confidence interval for m(x₀).     

On Confidence Intervals for Process Capability Indices in a One-Way Random Model

In this article, we investigated the bootstrap calibrated generalized confidence limits for process capability indices C _pk for the one-way random effect model. Also, we derived Bissell's approximation formula for the lower confidence limit using Satterthwaite's method and calculated its coverage probabilities and expected values. Then we compared it with standard bootstrap (SB) method and generalized confidence interval method. The simulation results indicate that the confidence limit obtained offers satisfactory coverage probabilities. The proposed method is illustrated with the help of simulation studies and data sets.     

MODEL SELECTION FOR PENALIZED SPLINE SMOOTHING USING AKAIKE INFORMATION CRITERIA

Two different forms of Akaike's information criterion (AIC) are compared for selecting the smooth terms in penalized spline additive mixed models. The conditional AIC (cAIC) has been used traditionally as a criterion for both estimating penalty parameters and selecting covariates in smoothing, and is based on the conditional likelihood given the smooth mean and on the effective degrees of freedom for a model fit. By comparison, the marginal AIC (mAIC) is based on the marginal likelihood from the mixed‐model formulation of penalized splines which has recently become popular for estimating smoothing parameters. To the best of the authors' knowledge, the use of mAIC for selecting covariates for smoothing in additive models is new. In the competing models considered for selection, covariates may have a nonlinear effect on the response, with the possibility of group‐specific curves. Simulations are used to compare the performance of cAIC and mAIC in model selection settings that have correlated and hierarchical smooth terms. In moderately large samples, both formulations of AIC perform extremely well at detecting the function that generated the data. The mAIC does better for simple functions, whereas the cAIC is more sensitive to detecting a true model that has complex and hierarchical terms.     

SELECTING "DEMONSTRABLY BEST" OR "DEMONSTRABLY WORST" EXPONENTIAL POPULATIONS

Consider k independent observations Y_i (i= 1,., k) from two-parameter exponential populations _i with location parameters μ and the same scale parameter If the μ_i are ranked as consider population as the “worst” population and II_p(k) as the “best” population (with some tagging so that p{) and p(k) are well defined in the case of equalities). If the Y_i are ranked as we consider the procedure, “Select provided Y_R(k) Y_r(k) is sufficiently large so that is demonstrably better than the other populations.” A similar procedure is studied for selecting the “demonstrably worst” population.     

Locking the Wiener Process to its Level-Crossing Time

We consider the specific transformation of a Wiener process {X(t), t ≥ 0} in the presence of an absorbing barrier a that results when this process is “time-locked” with respect to its first passage time T _a through a criterion level a, and the evolution of X(t) is considered backwards (retrospectively) from T _a . Formally, we study the random variables defined by Y(t) ≡ X(T _a  ? t) and derive explicit results for their density and mean, and also for their asymptotic forms. We discuss how our results can aid interpretations of time series “response-locked” to their times of crossing a criterion level.     

On derivaton and application of aic as a data-based criterion for histograms

In this paper a derivation of the Akaike's Information Criterion (AIC) is presented to select the number of bins of a histogram given only the data, showing that AIC strikes a balance between the “bias” and “variance” of the histogram estimate. Consistency of the criterion is discussed, an asymptotically optimal histogram bin width for the criterion is derived and its relationship to penalized likelihood methods is shown. A formula relating the optimal number of bins for a sample and a sub-sample obtained from it is derived. A number of numerical examples are presented.     

The effects of missing serial effects and/or heteroscedastic errors on mixed models using repeated growth data

When a two-level multilevel model (MLM) is used for repeated growth data, the individuals constitute level 2 and the successive measurements constitute level 1, which is nested within the individuals that make up level 2. The heterogeneity among individuals is represented by either the random-intercept or random-coefficient (slope) model. The variance components at level 1 involve serial effects and measurement errors under constant variance or heteroscedasticity. This study hypothesizes that missing serial effects or/and heteroscedasticity may bias the results obtained from two-level models. To illustrate this effect, we conducted two simulation studies, where the simulated data were based on the characteristics of an empirical mouse tumour data set. The results suggest that for repeated growth data with constant variance (measurement error) and misspecified serial effects (ρ > 0.3), the proportion of level-2 variation (intra-class correlation coefficient) increases with ρ and the two-level random-coefficient model is the minimum AIC (or AIC_c) model when compared with the fixed model, heteroscedasticity model, and random-intercept model. In addition, the serial effect (ρ > 0.1) and heteroscedasticity are both misspecified, implying that the two-level random-coefficient model is the minimum AIC (or AIC_c) model when compared with the fixed model and random-intercept model. This study demonstrates that missing serial effects and/or heteroscedasticity may indicate heterogeneity among individuals in repeated growth data (mixed or two-level MLM). This issue is critical in biomedical research.     

Planning for the 1950 Censuses of Population and Agriculture

The change from the z of “Student's” 1908 paper to the t of present day statistical theory and practice is traced and documented. It is shown that the change was brought about by the extension of “Student's” approach, by R.A. Fisher, to a broader class of problems, in response to a direct appeal from “Student” for a solution to one of these problems.     

On the information-based measure of covariance complexity and its application to the evaluation of multivariate linear models

This paper introduces a new information-theoretic measure of complexity called ICOMP as a decision rule for model selection and evaluation for multivariate linear models. The development of ICOMP is based on the generalization and utilization of the covariance complexity index of van Emden (1971) in estimation of the multivariate linear model. ICOMP is motivated by Akaike's (1973) Information Criterion (AIC), but it is a different procedure than AIC. In linear or nonlinear statistical models ICOMP uses an information-based characterization of: (i) the covariance matrix properties of the parameter estimates of a model starting from their finite sampling distributions, and (ii) the complexity of the inverse-Fisher information matrix (i-FIM) as a new criterion of achievable accuracy of the model As a result, it provides a trade-off between the accuracy of the parameter estimates and the interaction of the residuals of a model via the measure of complexity of their respective covariances. It controls the risks of both insufficient and overparameterized models, and incorporates the assumption of dependence and the independence of the residuals in one criterion function. A model with minimum ICOMP is chosen to be the best model among all possible competing alternative models. ICOMP relieves the researcher of any need to consider the parameter dimension of a model explicitly. A real numerical example is shown in subset selection of variables in multivariate regression analysis to demonstrate the utility and versatility of the new approach.     

ON THE CHOICE OF ESTIMATOR IN SURVERY SAMPLING*

We re-examine the criteria of “hyper-admissibility” and “necessary bestness”, for the choice of estimator, from the point of view of their relevance to the design of actual surveys. Both these criteria give rise to a unique choice of estimator (viz. the Horvitz-Thompson estimator ?_HT) whatever be the character under investigation or sample design. However, we show here that the “principal hyper-surfaces” (or “domains”) of dimension one (which are practically uninteresting)play the key role in arriving at the unique choice. A variance estimator v₁(?_HT) (due to Horvitz-Thompson), which takes negative values “often”, is shown to be uniquely “hyperadmissible” in a wide class of unbiased estimators of the variance of ?_HT. Extensive empirical evidence on the superiority of the Sen-Yates-Grundy variance estimator v₂(?_HT) over v₁(?_HT) is presented.     

Model selection for factor analysis: Some new criteria and performance comparisons

This paper derives Akaike information criterion (AIC), corrected AIC, the Bayesian information criterion (BIC) and Hannan and Quinn’s information criterion for approximate factor models assuming a large number of cross-sectional observations and studies the consistency properties of these information criteria. It also reports extensive simulation results comparing the performance of the extant and new procedures for the selection of the number of factors. The simulation results show the di?culty of determining which criterion performs best. In practice, it is advisable to consider several criteria at the same time, especially Hannan and Quinn’s information criterion, Bai and Ng’s IC_p2 and BIC₃, and Onatski’s and Ahn and Horenstein’s eigenvalue-based criteria. The model-selection criteria considered in this paper are also applied to Stock and Watson’s two macroeconomic data sets. The results differ considerably depending on the model-selection criterion in use, but evidence suggesting five factors for the first data and five to seven factors for the second data is obtainable.