Estimating Prediction Error: Cross-Validation vs. Accumulated Prediction Error

We study the validation of prediction rules such as regression models and classification algorithms through two out-of-sample strategies, cross-validation and accumulated prediction error. We use the framework of Efron (1983 Efron , B. ( 1983 ). Estimating the error rate of a prediction rule: improvement on cross-validation . Journal of the American Statistical Association 78 : 316 – 331 .Taylor & Francis Online], Web of Science ®] , Google Scholar]) where measures of prediction errors are defined as sample averages of expected errors and show through exact finite sample calculations that cross-validation and accumulated prediction error yield different smoothing parameter choices in nonparametric regression. The difference in choice does not vanish as sample size increases.