PREDICTION INTERVALS FOR SUPPORT VECTOR MACHINE REGRESSION

ABSTRACT

The support vector machine (SVM), first developed by Vapnik and his group at AT&T Bell Laboratories, is being used as a new technique for regression and classification problems. In this paper we present an approach to estimating prediction intervals for SVM regression based on posterior predictive densities. Furthermore, the method is illustrated with a data example.     

Sliced inverse regression under linear constraints

We consider the semiparametric regression model introduced by Li (1991) and add to this model some linear constraints on the slope parameters. These constraints can be identifiability conditions or they may carry additional in¬formations on the slope parameters. Using a geometric argument, we develop a method to estimate the slope parameters. This link-free and distribution-free method splits in two steps: the first is a Sliced Inverse Regression (SIR); Canonical Analysis is used at the second step to transform the SIR estimates so that they satisfy the constraints. We establish yn-consistency and obtain the asymptotic distribution of the estimates.

This estimation method is applied to the general sample selection model which is very useful in Econometrics. A simulation study shows that the method performs well in the example considered.     

A Bayesian Time Series Model of Multiple Structural Changes in Level,Trend, and Variance

We investigate whether seasonal-adjustment procedures are, at least approximately, linear data transformations. This question was initially addressed by Young and is important with respect to many issues including estimation of regression models with seasonally adjusted data. We focus on the X-11 program and rely on simulation evidence, involving linear unobserved component autoregressive integrated moving average models. We define a set of properties for the adequacy of a linear approximation to a seasonal-adjustment filter. These properties are examined through statistical tests. Next, we study the effect of X-11 seasonal adjustment on regression statistics assessing the statistical significance of the relationship between economic variables. Several empirical results involving economic data are also reported.     

On pseudo-values for regression analysis in competing risks models

For regression on state and transition probabilities in multi-state models Andersen et al. (Biometrika 90:15–27, 2003) propose a technique based on jackknife pseudo-values. In this article we analyze the pseudo-values suggested for competing risks models and prove some conjectures regarding their asymptotics (Klein and Andersen, Biometrics 61:223–229, 2005). The key is a second order von Mises expansion of the Aalen-Johansen estimator which yields an appropriate representation of the pseudo-values. The method is illustrated with data from a clinical study on total joint replacement. In the application we consider for comparison the estimates obtained with the Fine and Gray approach (J Am Stat Assoc 94:496–509, 1999) and also time-dependent solutions of pseudo-value regression equations.     

A Bayesian approach to non-parametric monotone function estimation

Summary.  The paper proposes two Bayesian approaches to non-parametric monotone function estimation. The first approach uses a hierarchical Bayes framework and a characterization of smooth monotone functions given by Ramsay that allows unconstrained estimation. The second approach uses a Bayesian regression spline model of Smith and Kohn with a mixture distribution of constrained normal distributions as the prior for the regression coefficients to ensure the monotonicity of the resulting function estimate. The small sample properties of the two function estimators across a range of functions are provided via simulation and compared with existing methods. Asymptotic results are also given that show that Bayesian methods provide consistent function estimators for a large class of smooth functions. An example is provided involving economic demand functions that illustrates the application of the constrained regression spline estimator in the context of a multiple-regression model where two functions are constrained to be monotone.     

Covariance-regularized regression and classification for high dimensional problems

Summary.  We propose covariance-regularized regression, a family of methods for prediction in high dimensional settings that uses a shrunken estimate of the inverse covariance matrix of the features to achieve superior prediction. An estimate of the inverse covariance matrix is obtained by maximizing the log-likelihood of the data, under a multivariate normal model, subject to a penalty; it is then used to estimate coefficients for the regression of the response onto the features. We show that ridge regression, the lasso and the elastic net are special cases of covariance-regularized regression, and we demonstrate that certain previously unexplored forms of covariance-regularized regression can outperform existing methods in a range of situations. The covariance-regularized regression framework is extended to generalized linear models and linear discriminant analysis, and is used to analyse gene expression data sets with multiple class and survival outcomes.     

Forecasting obesity trends in England

Summary.  Forecasts of trends in obesity in England for 2010 are produced by treating the available data, which contain the proportions of the population, categorized by age and sex, falling into different body mass index ranges, as compositional data sets, so that the implicit simplex restrictions are automatically satisfied. Forecasts are calculated by using linear trend models for the log-ratio transformations and are accompanied by prediction regions. The advantages of treating data on proportions compositionally are emphasized and compared with forecasts that have been obtained by ignoring this restriction.     

Estimating a sensitive proportion through randomized response procedures based on auxiliary information

Randomized response techniques are widely employed in surveys dealing with sensitive questions to ensure interviewee anonymity and reduce nonrespondents rates and biased responses. Since Warner’s (J Am Stat Assoc 60:63–69, 1965) pioneering work, many ingenious devices have been suggested to increase respondent’s privacy protection and to better estimate the proportion of people, π _A , bearing a sensitive attribute. In spite of the massive use of auxiliary information in the estimation of non-sensitive parameters, very few attempts have been made to improve randomization strategy performance when auxiliary variables are available. Moving from Zaizai’s (Model Assist Stat Appl 1:125–130, 2006) recent work, in this paper we provide a class of estimators for π _A , for a generic randomization scheme, when the mean of a supplementary non-sensitive variable is known. The minimum attainable variance bound of the class is obtained and the best estimator is also identified. We prove that the best estimator acts as a regression-type estimator which is at least as efficient as the corresponding estimator evaluated without allowing for the auxiliary variable. The general results are then applied to Warner and Simmons’ model.     

The t-information and its use in multivariate problems and time-series analysis

The second moment of recently introduced scalar inference function can be viewed as generalized Fisher information of the continuous probability distributions. In this paper we call it the t-information and give some possible applications of the new concept.     

Another look at Huber's estimator: A new minimax estimator in regression with stochastically bounded noise

Huber's estimator has had a long lasting impact, particularly on robust statistics. It is well known that under certain conditions, Huber's estimator is asymptotically minimax. A moderate generalization in rederiving Huber's estimator shows that Huber's estimator is not the only choice. We develop an alternative asymptotic minimax estimator and name it regression with stochastically bounded noise (RSBN). Simulations demonstrate that RSBN is slightly better in performance, although it is unclear how to justify such an improvement theoretically. We propose two numerical solutions: an iterative numerical solution, which is extremely easy to implement and is based on the proximal point method; and a solution by applying state-of-the-art nonlinear optimization software packages, e.g., SNOPT. Contribution: the generalization of the variational approach is interesting and should be useful in deriving other asymptotic minimax estimators in other problems.