Adaptive penalized quantile regression for high dimensional data

We propose a new adaptive L₁ penalized quantile regression estimator for high-dimensional sparse regression models with heterogeneous error sequences. We show that under weaker conditions compared with alternative procedures, the adaptive L₁ quantile regression selects the true underlying model with probability converging to one, and the unique estimates of nonzero coefficients it provides have the same asymptotic normal distribution as the quantile estimator which uses only the covariates with non-zero impact on the response. Thus, the adaptive L₁ quantile regression enjoys oracle properties. We propose a completely data driven choice of the penalty level _λn${\lambda }_n$, which ensures good performance of the adaptive L₁ quantile regression. Extensive Monte Carlo simulation studies have been conducted to demonstrate the finite sample performance of the proposed method.     

Improving the estimators of the parameters of a probit regression model: A ridge regression approach

This paper considered the estimation of the regression parameters of a general probit regression model. Accordingly, we proposed five ridge regression (RR) estimators for the probit regression models for estimating the parameters (β)$(\beta )$ when the weighted design matrix is ill-conditioned and it is suspected that the parameter β$\beta$ may belong to a linear subspace defined by Hβ=h$H\beta =h$. Asymptotic properties of the estimators are studied with respect to quadratic biases, MSE matrices and quadratic risks. The regions of optimality of the proposed estimators are determined based on the quadratic risks. Some relative efficiency tables and risk graphs are provided to illustrate the numerical comparison of the estimators. We conclude that when q≥3$q\ge 3$, one would uses PRRRE; otherwise one uses PTRRE with some optimum size α$\alpha$. We also discuss the performance of the proposed estimators compare to the alternative ridge regression method due to Liu (1993).     

Confidence intervals in regression utilizing prior information

We consider a linear regression model with regression parameter β=(_β1,…,_βp)$\beta =({\beta }_1,\dots ,{\beta }_p)$ and independent and identically N(0,σ²)$N(0,{\sigma }^2)$ distributed errors. Suppose that the parameter of interest is θ=a^Tβ$\theta =a^T\beta$ where a$a$ is a specified vector. Define the parameter τ=c^Tβ-t$\tau =c^T\beta -t$ where the vector c$c$ and the number t$t$ are specified and a$a$ and c$c$ are linearly independent. Also suppose that we have uncertain prior information that τ=0$\tau =0$. We present a new frequentist 1-α$1-\alpha$ confidence interval for θ$\theta$ that utilizes this prior information. We require this confidence interval to (a) have endpoints that are continuous functions of the data and (b) coincide with the standard 1-α$1-\alpha$ confidence interval when the data strongly contradict this prior information. This interval is optimal in the sense that it has minimum weighted average expected length where the largest weight is given to this expected length when τ=0$\tau =0$. This minimization leads to an interval that has the following desirable properties. This interval has expected length that (a) is relatively small when the prior information about τ$\tau$ is correct and (b) has a maximum value that is not too large. The following problem will be used to illustrate the application of this new confidence interval. Consider a 2×2$2\times 2$ factorial experiment with 20 replicates. Suppose that the parameter of interest θ$\theta$ is a specified simple   effect and that we have uncertain prior information that the two-factor interaction is zero. Our aim is to find a frequentist 0.95 confidence interval for θ$\theta$ that utilizes this prior information.     

Semiparametric additive isotonic regression

We consider the efficient estimation in the semiparametric additive isotonic regression model where each additive nonparametric component is assumed to be a monotone function. We show that the least-square estimator of the finite-dimensional regression coefficient is root-n$n$ consistent and asymptotically normal. Moreover, the isotonic estimator of each additive functional component is proved to have the oracle property, which means the additive component can be estimated with the highest asymptotic accuracy as if the other components were known. A fast algorithm is developed by iterating between a cyclic pool adjacent violators procedure and solving a standard ordinary least squares problem. Simulations are used to illustrate the performance of the proposed procedure and verify the oracle property.     

Goodness-of-fit tests for parametric regression with selection biased data

Consider the nonparametric location-scale regression model Y=m(X)+σ(X)ε$Y=m(X)+\sigma (X)\epsilon$, where the error ε$\epsilon$ is independent of the covariate X$X$, and m$m$ and σ$\sigma$ are smooth but unknown functions. The pair (X,Y)$(X,Y)$ is allowed to be subject to selection bias. We construct tests for the hypothesis that m(·)$m(·)$ belongs to some parametric family of regression functions. The proposed tests compare the nonparametric maximum likelihood estimator (NPMLE) based on the residuals obtained under the assumed parametric model, with the NPMLE based on the residuals obtained without using the parametric model assumption. The asymptotic distribution of the test statistics is obtained. A bootstrap procedure is proposed to approximate the critical values of the tests. Finally, the finite sample performance of the proposed tests is studied in a simulation study, and the developed tests are applied on environmental data.     

Quasi-minimax estimation in the general linear regression model

This paper is mainly concerned with minimax estimation in the general linear regression model y=Xβ+ε$y=X\beta +\epsilon$ under ellipsoidal restrictions on the parameter space and quadratic loss function. We confine ourselves to estimators that are linear in the response vector y  . The minimax estimators of the regression coefficient β$\beta$ are derived under homogeneous condition and heterogeneous condition, respectively. Furthermore, these obtained estimators are the ridge-type estimators and mean dispersion error (MDE) superior to the best linear unbiased estimator b=(X^′W^-1X)^-1X^′W^-1y$b=(X^{\prime }{W}^{-1}X{)}^{-1}{X}^{\prime }{W}^{-1}y$ under some conditions.     

On boosting kernel regression

In this paper we propose a simple multistep regression smoother which is constructed in an iterative manner, by learning the Nadaraya–Watson estimator with _L2$L_2$ boosting. We find, in both theoretical analysis and simulation experiments, that the bias converges exponentially fast, and the variance diverges exponentially slow. The first boosting step is analysed in more detail, giving asymptotic expressions as functions of the smoothing parameter, and relationships with previous work are explored. Practical performance is illustrated by both simulated and real data.     

Double-smoothing for bias reduction in local linear regression

Local linear regression involves fitting a straight line segment over a small region whose midpoint is the target point x, and the local linear estimate at x   is the estimated intercept of that straight line segment, with an asymptotic bias of order h²$h^2$ and variance of order (nh)^-1$(\mathit{nh}{)}^{-1}$ (h is the bandwidth). In this paper, we propose a new estimator, the double-smoothing local linear estimator, which is constructed by integrally combining all fitted values at x   of local lines in its neighborhood with another round of smoothing. The proposed estimator attempts to make use of all information obtained from fitting local lines. Without changing the order of variance, the new estimator can reduce the bias to an order of h⁴$h^4$. The proposed estimator has better performance than local linear regression in situations with considerable bias effects; it also has less variability and more easily overcomes the sparse data problem than local cubic regression. At boundary points, the proposed estimator is comparable to local linear regression. Simulation studies are conducted and an ethanol example is used to compare the new approach with other competitive methods.     

Semiparametric partially linear regression models for functional data

In the context of longitudinal data analysis, a random function typically represents a subject that is often observed at a small number of time point. For discarding this restricted condition of observation number of each subject, we consider the semiparametric partially linear regression models with mean function x^?β$x^{?}\beta$ + g(z), where x and z   are functional data. The estimations of β$\beta$ and g(z) are presented and some asymptotic results are given. It is shown that the estimator of the parametric component is asymptotically normal. The convergence rate of the estimator of the nonparametric component is also obtained. Here, the observation number of each subject is completely flexible. Some simulation study is conducted to investigate the finite sample performance of the proposed estimators.     

Robust nonparametric estimation for spatial regression

In this paper, we investigate a nonparametric robust estimation for spatial regression. More precisely, given a strictly stationary random field _Zi=(_Xi,_{Yi)i∈NNN≥1}$Z_{\mathbf{i}}=(X_{\mathbf{i}},Y_{\mathbf{i}}{)}_{\mathbf{i}\in {\mathbb{N}}^{N}N\ge 1}$, we consider a family of robust nonparametric estimators for a regression function based on the kernel method. Under some general mixing assumptions, the almost complete consistency and the asymptotic normality of these estimators are obtained. A robust procedure to select the smoothing parameter adapted to the spatial data is also discussed.     

Estimation of smooth regression functions in monotone response models

We consider the estimation of smooth regression functions in a class of conditionally parametric co-variate-response models. Independent and identically distributed observations are available from the distribution of (Z,X)$(Z,X)$, where Z is a real-valued co-variate with some unknown distribution, and the response X conditional on Z   is distributed according to the density p(·,ψ(Z))$p(·,\psi (Z))$, where p(·,θ)$p(·,\theta )$ is a one-parameter exponential family. The function ψ$\psi$ is a smooth monotone function. Under this formulation, the regression function E(X|Z)$E(X|Z)$ is monotone in the co-variate Z   (and can be expressed as a one–one function of ψ$\psi$); hence the term “monotone response model”. Using a penalized least squares approach that incorporates both monotonicity and smoothness, we develop a scheme for producing smooth monotone estimates of the regression function and also the function ψ$\psi$ across this entire class of models. Point-wise asymptotic normality of this estimator is established, with the rate of convergence depending on the smoothing parameter. This enables construction of Wald-type (point-wise) as well as pivotal confidence sets for ψ$\psi$ and also the regression function. The methodology is extended to the general heteroscedastic model, and its asymptotic properties are discussed.     

Learning rates of regularized regression for exponentially strongly mixing sequence

The study of regularized learning algorithms associated with least squared loss is one of very important issues. Wu et al. [2006. Learning rates of least-square regularized regression. Found. Comput. Math. 6, 171–192] established fast learning rates m^-θ$m^{-\theta }$ for the least square regularized regression in reproducing kernel Hilbert spaces under some assumptions on Mercer kernels and on regression functions, where m   denoted the number of the samples and θ$\theta$ may be arbitrarily close to 1. They assumed as in most existing works that the set of samples were drawn independently from the underlying probability. However, independence is a very restrictive concept. Without the independence of samples, the study of learning algorithms is more involved, and little progress has been made. The aim of this paper is to establish the above results of Wu et al. for the dependent samples. The dependence of samples in this paper is expressed in terms of exponentially strongly mixing sequence.     

Robust nonparametric estimation with missing data

In this paper, under a nonparametric regression model, we introduce two families of robust procedures to estimate the regression function when missing data occur in the response. The first proposal is based on a local M$M$-functional applied to the conditional distribution function estimate adapted to the presence of missing data. The second proposal imputes the missing responses using the local M$M$-smoother based on the observed sample and then estimates the regression function with the completed sample. We show that the robust procedures considered are consistent and asymptotically normally distributed. A robust procedure to select the smoothing parameter is also discussed.     

Optimal global rates of convergence for nonparametric regression with unbounded data

Estimation of regression functions from independent and identically distributed data is considered. The _L2$L_2$ error with integration with respect to the design measure is used as an error criterion. Usually in the analysis of the rate of convergence of estimates a boundedness assumption on the explanatory variable X$X$ is made besides smoothness assumptions on the regression function and moment conditions on the response variable Y$Y$. In this article we consider the kernel estimate and show that by replacing the boundedness assumption on X$X$ by a proper moment condition the same (optimal) rate of convergence can be shown as for bounded data. This answers Question 1 in Stone [1982. Optimal global rates of convergence for nonparametric regression. Ann. Statist., 10, 1040–1053].     

Collapsibility of regression coefficients and its extensions

Collapsibility with respect to a measure of association implies that the measure of association can be obtained from the marginal model. We first discuss model collapsibility and collapsibility with respect to regression coefficients for linear regression models. For parallel regression models, we give simple and different proofs of some of the known results and obtain also certain new results. For random coefficient regression models, we define (average) A$A$-collapsibility and obtain conditions under which it holds. We consider Poisson regression and logistic regression models also, and derive conditions for collapsibility and A$A$-collapsibility, respectively. These results generalize some of the results available in the literature. Some suitable examples are also discussed.     

A test for the parametric form of the variance function in a partial linear regression model

We consider the problem of testing for a parametric form of the variance function in a partial linear regression model. A new test is derived, which can detect local alternatives converging to the null hypothesis at a rate n^-1/2$n^{-1/2}$ and is based on a stochastic process of the integrated variance function. We establish weak convergence to a Gaussian process under the null hypothesis, fixed and local alternatives. In the special case of testing for homoscedasticity the limiting process is a scaled Brownian bridge. We also compare the finite sample properties with a test based on an L²$L^2$-distance, which was recently proposed by You and Chen [2005. Testing heteroscedasticity in partially linear regression models. Statist. Probab. Lett. 73, 61–70].     

Optimal designs based on exact confidence regions for parameter estimation of a nonlinear regression model

This paper is concerned with the proposal of optimality criteria, referred to as X  - and X^′$X^{\prime }$-optimality criteria, and the construction of X  - and X^′$X^{\prime }$-optimal designs, for nonlinear regression models. These optimal designs aim at improving the estimation of parameters of this class of models. The principle of these criteria is the minimization, with respect to the design, of the expected volume of a particular exact parametric confidence region. In this paper we give detailed definitions, properties, and computation methods of X  - and X^′$X^{\prime }$-optimal designs. We also compare these designs with the classic local D-optimal designs, with regard to robustness and efficiency, for two very well-known academic models (Box–Lucas and Michaelis–Menten models).