A model-averaging approach for smoothing spline regression

ABSTRACT

This article considers nonparametric regression problems and develops a model-averaging procedure for smoothing spline regression problems. Unlike most smoothing parameter selection studies determining an optimum smoothing parameter, our focus here is on the prediction accuracy for the true conditional mean of Y given a predictor X. Our method consists of two steps. The first step is to construct a class of smoothing spline regression models based on nonparametric bootstrap samples, each with an appropriate smoothing parameter. The second step is to average bootstrap smoothing spline estimates of different smoothness to form a final improved estimate. To minimize the prediction error, we estimate the model weights using a delete-one-out cross-validation procedure. A simulation study has been performed by using a program written in R. The simulation study provides a comparison of the most well known cross-validation (CV), generalized cross-validation (GCV), and the proposed method. This new method is straightforward to implement, and gives reliable performances in simulations.     

Concentration inequalities of the cross-validation estimator for empirical risk minimizer

ABSTRACT

We derive concentration inequalities for the cross-validation estimate of the generalization error for empirical risk minimizers. In the general setting, we show that the worst-case error of this estimate is not much worse that of training error estimate see Kearns M, Ron D. [Algorithmic stability and sanity-check bounds for leave-one-out cross-validation. Neural Comput. 1999;11:1427–1453]. General loss functions and class of predictors with finite VC-dimension are considered. Our focus is on proving the consistency of the various cross-validation procedures. We point out the interest of each cross-validation procedure in terms of rates of convergence. An interesting consequence is that the size of the test sample is not required to grow to infinity for the consistency of the cross-validation procedure.     

Empirical Performance of Cross-Validation With Oracle Methods in a Genomics Context

When employing model selection methods with oracle properties such as the smoothly clipped absolute deviation (SCAD) and the Adaptive Lasso, it is typical to estimate the smoothing parameter by m-fold cross-validation, for example, m = 10. In problems where the true regression function is sparse and the signals large, such cross-validation typically works well. However, in regression modeling of genomic studies involving Single Nucleotide Polymorphisms (SNP), the true regression functions, while thought to be sparse, do not have large signals. We demonstrate empirically that in such problems, the number of selected variables using SCAD and the Adaptive Lasso, with 10-fold cross-validation, is a random variable that has considerable and surprising variation. Similar remarks apply to non-oracle methods such as the Lasso. Our study strongly questions the suitability of performing only a single run of m-fold cross-validation with any oracle method, and not just the SCAD and Adaptive Lasso.     

Estimation Bias in the First-Order Autoregressive Model and Its Impact on Predictions and Prediction Intervals

The least squares estimate of the autoregressive coefficient in the AR(1) model is known to be biased towards zero, especially for parameters close to the stationarity boundary. Several methods for correcting the autoregressive parameter estimate for the bias have been suggested. Using simulations, we study the bias and the mean square error of the least squares estimate and the bias-corrections proposed by Kendall and Quenouille.

We also study the mean square forecast error and the coverage of the 95% prediction interval when using the biased least squares estimate or one of its bias-corrected versions. We find that the estimation bias matters little for point forecasts, but that it affects the coverage of the prediction intervals. Prediction intervals for forecasts more than one step ahead, when calculated with the biased least squares estimate, are too narrow.     

The conditional cumulative distribution function in single functional index model

ABSTRACT

We consider the estimation of the conditional cumulative distribution function of a scalar response variable Y given a Hilbertian random variable X when the observations are linked via a single-index structure. We establish the pointwise and the uniform almost complete convergence (with the rate) of the kernel estimate of this model. As an application, we show how our result can be applied in the prediction problem via the conditional median estimate. Also, the choice of the functional index via the cross-validation procedure is also discussed but not attacked.     

Subsampling-extrapolation bandwidth selection in bivariate kernel density estimation

This paper focuses on bivariate kernel density estimation that bridges the gap between univariate and multivariate applications. We propose a subsampling-extrapolation bandwidth matrix selector that improves the reliability of the conventional cross-validation method. The proposed procedure combines a U-statistic expression of the mean integrated squared error and asymptotic theory, and can be used in both cases of diagonal bandwidth matrix and unconstrained bandwidth matrix. In the subsampling stage, one takes advantage of the reduced variability of estimating the bandwidth matrix at a smaller subsample size m (m < n); in the extrapolation stage, a simple linear extrapolation is used to remove the incurred bias. Simulation studies reveal that the proposed method reduces the variability of the cross-validation method by about 50% and achieves an expected integrated squared error that is up to 30% smaller than that of the benchmark cross-validation. It shows comparable or improved performance compared to other competitors across six distributions in terms of the expected integrated squared error. We prove that the components of the selected bivariate bandwidth matrix have an asymptotic multivariate normal distribution, and also present the relative rate of convergence of the proposed bandwidth selector.     

中国股票市场的长期记忆性与趋势预测研究

本文首次在国内利用较新的精准局部似然函数法（Exact Local Whittle）,以上证指数为对象,估计了ARFIMA(p,d,q)模型的长期记忆参数d,并分析了上证指数的趋势性变化。估计结果和稳健性检验均表明,上证指数具有长期记忆性,以上证指数为代表的股票市场并非有效;模拟结果显示,当滚动窗口n=260,带宽m=[n0.65]时,长期记忆参数即估计量d既具备一致性,又具有渐进正态性。在2004年10月8日至2015年11月13日期间,模型给出了8次上涨或下跌的趋势转换信号,其中7次信号是正确的,仅有1次给出了错误信号;股票价格由下跌趋势转为上涨趋势、由上涨趋势转为下跌趋势两种情况相比,记忆参数d在前一种情况时下跌幅度更大,给出的信号更明显。     

Estimating prediction error for complex samples

With a growing interest in using non-representative samples to train prediction models for numerous outcomes it is necessary to account for the sampling design that gives rise to the data in order to assess the generalized predictive utility of a proposed prediction rule. After learning a prediction rule based on a non-uniform sample, it is of interest to estimate the rule's error rate when applied to unobserved members of the population. Efron (1986) proposed a general class of covariance penalty inflated prediction error estimators that assume the available training data are representative of the target population for which the prediction rule is to be applied. We extend Efron's estimator to the complex sample context by incorporating Horvitz–Thompson sampling weights and show that it is consistent for the true generalization error rate when applied to the underlying superpopulation. The resulting Horvitz–Thompson–Efron estimator is equivalent to dAIC, a recent extension of Akaike's information criteria to survey sampling data, but is more widely applicable. The proposed methodology is assessed with simulations and is applied to models predicting renal function obtained from the large-scale National Health and Nutrition Examination Study survey. The Canadian Journal of Statistics 48: 204–221; 2020 © 2019 Statistical Society of Canada     

Conditional covariance penalties for mixed models

The prediction error for mixed models can have a conditional or a marginal perspective depending on the research focus. We introduce a novel conditional version of the optimism theorem for mixed models linking the conditional prediction error to covariance penalties for mixed models. Different possibilities for estimating these conditional covariance penalties are introduced. These are bootstrap methods, cross-validation, and a direct approach called Steinian. The behavior of the different estimation techniques is assessed in a simulation study for the binomial-, the t-, and the gamma distribution and for different kinds of prediction error. Furthermore, the impact of the estimation techniques on the prediction error is discussed based on an application to undernutrition in Zambia.     

An MSE‐reduced estimator for the response proportion in a two‐stage clinical trial

Two‐stage design is very useful in clinical trials for evaluating the validity of a specific treatment regimen. When the second stage is allowed to continue, the method used to estimate the response rate based on the results of both stages is critical for the subsequent design. The often‐used sample proportion has an evident upward bias. However, the maximum likelihood estimator or the moment estimator tends to underestimate the response rate. A mean‐square error weighted estimator is considered here; its performance is thoroughly investigated via Simon's optimal and minimax designs and Shuster's design. Compared with the sample proportion, the proposed method has a smaller bias, and compared with the maximum likelihood estimator, the proposed method has a smaller mean‐square error. Copyright © 2010 John Wiley & Sons, Ltd.     

Smooth tests of fit for censored gmma samples

In this paper properties of two estimators of Cpm are investigated in terms of changes in the process mean and variance. The bias and mean squared error of these estimators are derived. It can be shown that the estimate of Cpm proposed by Chan, Cheng and Spiring (1988) has smaller bias than the one proposed by Boyles (1991) and also has a smaller mean squared error under certain conditions. Various approximate confidence intervals for Cpm are obtained and are compared in terms of coverage probabilities, missed rate and average interval width.     

Interest-Rate Arbitrage in Currency Baskets: Forecasting Weights and Measuring Risk

This article addresses the problem of the bias of income and expenditure elasticities estimated on pseudopanel data caused by measurement error and unobserved heterogeneity. We gauge these biases empirically by comparing cross-sectional, pseudo-panel, and true panel data from both Polish and U.S. expenditure surveys. Our results suggest that unobserved heterogeneity imparts a downward bias to cross-section estimates of income elasticities of at-home food expenditures and an upward bias to estimates of income elasticities of away-from-home food expenditures. “Within” and first-difference estimators suffer less bias, but only if the effects of measurement error are accounted for with instrumental variables.     

AN ASYMPTOTIC EXPANSION OF THE EXPECTATION OF THE ESTIMATED ERROR RATE IN DISCRIMINANT ANALYSIS1

When a sample discriminant function is computed, it is desired to estimate the error rate using this function. This is often done by computing G(-D/2), where G is the cumulative normal distribution and D² is the estimated Mahalanobis' distance. In this paper an asymptotic expansion of the expectation of G(-D/2) is derived and is compared with existing Monte Carlo estimates. The asymptotic bias of G(-D/2) is derived also and the well-known practical result that G(-D/2) gives too favourable an estimate of the true error rate     

Ideal bootstrap estimation of expected prediction error for k-nearest neighbor classifiers: Applications for classification and error assessment

Euclidean distance k-nearest neighbor (k-NN) classifiers are simple nonparametric classification rules. Bootstrap methods, widely used for estimating the expected prediction error of classification rules, are motivated by the objective of calculating the ideal bootstrap estimate of expected prediction error. In practice, bootstrap methods use Monte Carlo resampling to estimate the ideal bootstrap estimate because exact calculation is generally intractable. In this article, we present analytical formulae for exact calculation of the ideal bootstrap estimate of expected prediction error for k-NN classifiers and propose a new weighted k-NN classifier based on resampling ideas. The resampling-weighted k-NN classifier replaces the k-NN posterior probability estimates by their expectations under resampling and predicts an unclassified covariate as belonging to the group with the largest resampling expectation. A simulation study and an application involving remotely sensed data show that the resampling-weighted k-NN classifier compares favorably to unweighted and distance-weighted k-NN classifiers.     

Decomposition of Prediction Error in Multilevel Models

ABSTRACT

We present a decomposition of prediction error for the multilevel model in the context of predicting a future observable y _*j in the jth group of a hierarchical dataset. The multilevel prediction rule is used for prediction and the components of prediction error are estimated via a simulation study that spans the various combinations of level-1 (individual) and level-2 (group) sample sizes and different intraclass correlation values. Additionally, analytical results present the increase in predicted mean square error (PMSE) with respect to prediction error bias. The components of prediction error provide information with respect to the cost of parameter estimation versus data imputation for predicting future values in a hierarchical data set. Specifically, the cost of parameter estimation is very small compared to data imputation.     

Computationally efficient classes of higher-order kernel functions

Classes of higher-order kernels for estimation of a probability density are constructed by iterating the twicing procedure. Given a kernel K of order l, we build a family of kernels K_m of orders l(m + 1) with the attractive property that their Fourier transforms are simply 1 — {1 —$(.)}^m+1, where ? is the Fourier transform of K. These families of higher-order kernels are well suited when the fast Fourier transform is used to speed up the calculation of the kernel estimate or the least-squares cross-validation procedure for selection of the window width. We also compare the theoretical performance of the optimal polynomial-based kernels with that of the iterative twicing kernels constructed from some popular second-order kernels.     

Transformation and Smoothing in Sample Survey Data

Abstract. We consider model‐based prediction of a finite population total when a monotone transformation of the survey variable makes it appropriate to assume additive, homoscedastic errors. As the transformation to achieve this does not necessarily simultaneously produce an easily parameterized mean function, we assume only that the mean is a smooth function of the auxiliary variable and estimate it non‐parametrically. The back transformation of predictions obtained on the transformed scale introduces bias which we remove using smearing. We obtain an asymptotic expansion for the prediction error which shows that prediction bias is asymptotically negligible and the prediction mean‐squared error (MSE) using a non‐parametric model remains in the same order as when a parametric model is adopted. The expansion also shows the effect of smearing on the prediction MSE and can be used to compute the asymptotic prediction MSE. We propose a model‐based bootstrap estimate of the prediction MSE. The predictor produces competitive results in terms of bias and prediction MSE in a simulation study, and performs well on a population constructed from an Australian farm survey.     

New ASA Fellows — 1964

This is an invited expository article for The American Statistician. It reviews the nonparametric estimation of statistical error, mainly the bias and standard error of an estimator, or the error rate of a prediction rule. The presentation is written at a relaxed mathematical level, omitting most proofs, regularity conditions, and technical details.     

Bayes and Empirical Bayes Inference in Changepoint Problems

ABSTRACT

In the case of the random design nonparametric regression, the double smoothing technique is applied to estimate the multivariate regression function. The proposed estimator has desirable properties in both the finite sample and the asymptotic cases. In the finite sample case, it has bounded conditional (and unconditional) bias and variance. On the other hand, in the asymptotic case, it has the same mean square error as the local linear estimator in Fan (Design-Adaptive Nonparametric Regression. Journal of the American Statistical Association 1992, 87, 998–1004; Local Linear Regression Smoothers and Their Minimax Efficiencies. Annals of Statistics 1993, 21, 196–216). Simulation studies demonstrate that the proposed estimator is better than the local linear estimator, because it has a smaller sample mean integrated square error and gives smoother estimates.