Efficient Bandwidth Selection in Non-parametric Regression

In this paper we use non-parametric local polynomial methods to estimate the regression function, m ( x ). Y may be a binary or continuous response variable, and X is continuous with non-uniform density. The main contributions of this paper are the weak convergence of a bandwidth process for kernels of order (0, k ), k =2 ^j , j ≥1 and the proposal of a local data-driven bandwidth selection method which is particularly beneficial for the case when X is not distributed uniformly. This selection method minimizes estimates of the asymptotic MSE and estimates the bias portion in an innovative way which relies on the order of the kernel and not estimation of m ²( x ) directly. We show that utilization of this method results in the achievement of the optimal asymptotic MSE by the estimator, i.e. the method is efficient. Simulation studies are provided which illustrate the method for both binary and continuous response cases.     

Estimating smooth monotone functions

Many situations call for a smooth strictly monotone function f of arbitrary flexibility. The family of functions defined by the differential equation D  ² f  = w Df , where w is an unconstrained coefficient function comprises the strictly monotone twice differentiable functions. The solution to this equation is f = C ₀ + C ₁  D ⁻¹{exp( D ⁻¹ w )}, where C ₀ and C ₁ are arbitrary constants and D ⁻¹ is the partial integration operator. A basis for expanding w is suggested that permits explicit integration in the expression of f . In fitting data, it is also useful to regularize f by penalizing the integral of w ² since this is a measure of the relative curvature in f . Applications are discussed to monotone nonparametric regression, to the transformation of the dependent variable in non-linear regression and to density estimation.     

Beta-Bernstein Smoothing for Regression Curves with Compact Support

ABSTRACT. The problem of boundary bias is associated with kernel estimation for regression curves with compact support. This paper proposes a simple and uni(r)ed approach for remedying boundary bias in non-parametric regression, without dividing the compact support into interior and boundary areas and without applying explicitly different smoothing treatments separately. The approach uses the beta family of density functions as kernels. The shapes of the kernels vary according to the position where the curve estimate is made. Theyare symmetric at the middle of the support interval, and become more and more asymmetric nearer the boundary points. The kernels never put any weight outside the data support interval, and thus avoid boundary bias. The method is a generalization of classical Bernstein polynomials, one of the earliest methods of statistical smoothing. The proposed estimator has optimal mean integrated squared error at an order of magnitude n ^−4/5, equivalent to that of standard kernel estimators when the curve has an unbounded support.     

Empirical Likelihood-based Inference in Linear Models with Missing Data

The missing response problem in linear regression is studied. An adjusted empirical likelihood approach to inference on the mean of the response variable is developed. A non-parametric version of Wilks's theorem for the adjusted empirical likelihood is proved, and the corresponding empirical likelihood confidence interval for the mean is constructed. With auxiliary information, an empirical likelihood-based estimator with asymptotic normality is defined and an adjusted empirical log-likelihood function with asymptotic χ² is derived. A simulation study is conducted to compare the adjusted empirical likelihood methods and the normal approximation methods in terms of coverage accuracies and average lengths of the confidence intervals. Based on biases and standard errors, a comparison is also made between the empirical likelihood-based estimator and related estimators by simulation. Our simulation indicates that the adjusted empirical likelihood methods perform competitively and the use of auxiliary information provides improved inferences.     

Estimation and inference in regression discontinuity designs with asymmetric kernels

We study the behaviour of the Wald estimator of causal effects in regression discontinuity design when local linear regression (LLR) methods are combined with an asymmetric gamma kernel. We show that the resulting statistic is no more complex to implement than existing methods, remains consistent at the usual non-parametric rate, and maintains an asymptotic normal distribution but, crucially, has bias and variance that do not depend on kernel-related constants. As a result, the new estimator is more efficient and yields more reliable inference. A limited Monte Carlo experiment is used to illustrate the efficiency gains. As a by product of the main discussion, we extend previous published work by establishing the asymptotic normality of the LLR estimator with a gamma kernel. Finally, the new method is used in a substantive application.     

Non-parametric Kernel Estimation of the Coefficient of a Diffusion

In this work we exhibit a non-parametric estimator of kernel type, for the diffusion coefficient when one observes a one-dimensional diffusion process at times i / n for i = , ..., n and study its asymptotics as n ←∞. When the diffusion coefficient has regularity r ≥ 1, we obtain a rate 1/ n ^{r /(1+2 r )}, both for pointwise estimation and for estimation on a compact subset of R: this is the same rate as for non-parametric estimation of a density with i.i.d. observations.     

Likelihood Ratio Tests Under Local and Fixed Alternatives in Monotone Function Problems

Abstract.  We focus on a class of non-standard problems involving non-parametric estimation of a monotone function that is characterized by n ^1/3 rate of convergence of the maximum likelihood estimator, non-Gaussian limit distributions and the non-existence of     -regular estimators. We have shown elsewhere that under a null hypothesis of the type ψ ( z ₀) =  θ ₀ ( ψ being the monotone function of interest) in non-standard problems of the above kind, the likelihood ratio statistic has a 'universal' limit distribution that is free of the underlying parameters in the model. In this paper, we illustrate its limiting behaviour under local alternatives of the form ψ _n ( z ), where ψ _n (·) and ψ (·) vary in O ( n ^−1/3) neighbourhoods around z ₀ and ψ _n converges to ψ at rate n ^1/3 in an appropriate metric. Apart from local alternatives, we also consider the behaviour of the likelihood ratio statistic under fixed alternatives and establish the convergence in probability of an appropriately scaled version of the same to a constant involving a Kullback–Leibler distance.     

On distribution-weighted partial least squares with diverging number of highly correlated predictors

Summary.  Because highly correlated data arise from many scientific fields, we investigate parameter estimation in a semiparametric regression model with diverging number of predictors that are highly correlated. For this, we first develop a distribution-weighted least squares estimator that can recover directions in the central subspace, then use the distribution-weighted least squares estimator as a seed vector and project it onto a Krylov space by partial least squares to avoid computing the inverse of the covariance of predictors. Thus, distrbution-weighted partial least squares can handle the cases with high dimensional and highly correlated predictors. Furthermore, we also suggest an iterative algorithm for obtaining a better initial value before implementing partial least squares. For theoretical investigation, we obtain strong consistency and asymptotic normality when the dimension p of predictors is of convergence rate O { n ^1/2/ log ( n )} and o ( n ^1/3) respectively where n is the sample size. When there are no other constraints on the covariance of predictors, the rates n ^1/2 and n ^1/3 are optimal. We also propose a Bayesian information criterion type of criterion to estimate the dimension of the Krylov space in the partial least squares procedure. Illustrative examples with a real data set and comprehensive simulations demonstrate that the method is robust to non-ellipticity and works well even in 'small n –large p ' problems.     

An Adaptive Two-stage Estimation Method for Additive Models

Abstract.  In this paper, a two-stage estimation method for non-parametric additive models is investigated. Differing from Horowitz and Mammen's two-stage estimation, our first-stage estimators are designed not only for dimension reduction but also as initial approximations to all of the additive components. The second-stage estimators are obtained by using one-dimensional non-parametric techniques to refine the first-stage ones. From this procedure, we can reveal a relationship between the regression function spaces and convergence rate, and then provide estimators that are optimal in the sense that, better than the usual one-dimensional mean-squared error (MSE) of the order n ^−4/5 , the MSE of the order n ^{− 1} can be achieved when the underlying models are actually parametric. This shows that our estimation procedure is adaptive in a certain sense. Also it is proved that the bandwidth that is selected by cross-validation depends only on one-dimensional kernel estimation and maintains the asymptotic optimality. Simulation studies show that the new estimators of the regression function and all components outperform the existing estimators, and their behaviours are often similar to that of the oracle estimator.     

Parametrically Guided Non-parametric Regression

We present a new approach to regression function estimation in which a non-parametric regression estimator is guided by a parametric pilot estimate with the aim of reducing the bias. New classes of parametrically guided kernel weighted local polynomial estimators are introduced and formulae for asymptotic expectation and variance, hence approximated mean squared error and mean integrated squared error, are derived. It is shown that the new classes of estimators have the very same large sample variance as the estimators in the standard non-parametric setting, while there is substantial room for reducing the bias if the chosen parametric pilot function belongs to a wide neighbourhood around the true regression line. Bias reduction is discussed in light of examples and simulations.     

Correction of Density Estimators that are not Densities

Abstract. Several old and new density estimators may have good theoretical performance, but are hampered by not being bona fide densities; they may be negative in certain regions or may not integrate to 1. One can therefore not simulate from them, for example. This paper develops general modification methods that turn any density estimator into one which is a bona fide density, and which is always better in performance under one set of conditions and arbitrarily close in performance under a complementary set of conditions. This improvement-for-free procedure can, in particular, be applied for higher-order kernel estimators, classes of modern h ⁴ bias kernel type estimators, superkernel estimators, the sinc kernel estimator, the k -NN estimator, orthogonal expansion estimators, and for various recently developed semi-parametric density estimators.     

On the Distance Between Cumulative Sum Diagram and Its Greatest Convex Minorant for Unequally Spaced Design Points

Abstract.  The supremum difference between the cumulative sum diagram, and its greatest convex minorant (GCM), in case of non-parametric isotonic regression is considered. When the regression function is strictly increasing, and the design points are unequally spaced, but approximate a positive density in even a slow rate ( n ^−1/3), then the difference is shown to shrink in a very rapid (close to n ^−2/3) rate. The result is analogous to the corresponding result in case of a monotone density estimation established by Kiefer and Wolfowitz, but uses entirely different representation. The limit distribution of the GCM as a process on the unit interval is obtained when the design variables are i.i.d. with a positive density. Finally, a pointwise asymptotic normality result is proved for the smooth monotone estimator, obtained by the convolution of a kernel with the classical monotone estimator.     

Penalized Pseudolikelihood Inference in Spatial Interaction Models with Covariates

Given spatially located observed random variables ( x , z = {( x _i , z _i )} _i , we propose a new method for non-parametric estimation of the potential functions of a Markov random field p ( x | z ), based on a roughness penalty approach. The new estimator maximizes the penalized log-pseudolikelihood function and is a natural cubic spline. The calculations involved do not rely on Monte Carlo simulation. We suggest the use of B-splines to stabilize the numerical procedure. An application in Bayesian image reconstruction is described.     

Non-parametric Regression with Dependent Censored Data

Abstract.  Let ( X _i , Y _i ) ( i = 1 ,…, n ) be n replications of a random vector ( X , Y  ), where Y is supposed to be subject to random right censoring. The data ( X _i , Y _i ) are assumed to come from a stationary α -mixing process. We consider the problem of estimating the function m ( x ) = E ( φ ( Y ) |  X = x ), for some known transformation φ . This problem is approached in the following way: first, we introduce a transformed variable     , that is not subject to censoring and satisfies the relation     , and then we estimate m ( x ) by applying local linear regression techniques. As a by-product, we obtain a general result on the uniform rate of convergence of kernel type estimators of functionals of an unknown distribution function, under strong mixing assumptions.     

Estimating the Upper Support Point in Deconvolution

Abstract.  We consider estimation of the upper boundary point F ⁻¹ (1) of a distribution function F with finite upper boundary or 'frontier' in deconvolution problems, primarily focusing on deconvolution models where the noise density is decreasing on the positive halfline. Our estimates are based on the (non-parametric) maximum likelihood estimator (MLE) of F . We show that (1) is asymptotically never too small. If the convolution kernel has bounded support the estimator (1) can generally be expected to be consistent. In this case, we establish a relation between the extreme value index of F and the rate of convergence of (1) to the upper support point for the 'boxcar' deconvolution model. If the convolution density has unbounded support, (1) can be expected to overestimate the upper support point. We define consistent estimators , for appropriately chosen vanishing sequences ( β _n ) and study these in a particular case.     

Rejoinder to 'Ahmed, M.S. (1998). A note on regression-type estimators using multiple auxiliary information.'

In the estimators t ₃ , t ₄ , t ₅ of Mukerjee, Rao & Vijayan (1987), b _{y x} and b _{y z} are partial regression coefficients of y on x and z , respectively, based on the smaller sample. With the above interpretation of b _{y x} and b _{y z} in t ₃ , t ₄ , t ₅ , all the calculations in Mukerjee at al. (1987) are correct. In this connection, we also wish to make it explicit that b _{x z} in t ₅ is an ordinary and not a partial regression coefficient. The 'corrected' MSEs of t ₃ , t ₄ , t ₅ , as given in Ahmed (1998 Section 3) are computed assuming that our b _{y x} and b _{y z} are ordinary and not partial regression coefficients. Indeed, we had no intention of giving estimators using the corresponding ordinary regression coefficients which would lead to estimators inferior to those given by Kiregyera (1984). We accept responsibility for any notational confusion created by us and express regret to readers who have been confused by our notation. Finally, in consideration of the above, it may be noted that Tripathi & Ahmed's (1995) estimator t ₀ , quoted also in Ahmed (1998), is no better than t ₅ of Mukerjee at al. (1987).     

A TWO-PHASE SAMPLING ESTIMATOR IN SAMPLE SURVEYS

A two-phase sampling estimator of the ratio-type for estimating the mean of a finite population, has been considered where the value of ρC_y/C_x can be guessed or estimated in advance. Here C_y and C_x denote respectively the coefficients of variation of the characteristic under study, y, and the auxiliary characteristic x and ρ denotes the coefficient of correlation between y and x. When the value of ρC_y/C_x is guessed or estimated exactly, the estimator has a smaller large-sample variance compared with either an ordinary ratio estimator or an ordinary linear regression estimator in two-phase sampling in the case where the first-phase sample is drawn independently from the second-phase sample. If the sample at the second phase is a subsample of the first-phase sample, the estimator has variance equal to that of the linear regression estimator. The largest value of the difference between the assumed value and the actual value of ρC_y/C_x has been obtained so as not to result in the variance of the estimator being larger than the variances of either an ordinary ratio estimator or an ordinary linear regression estimator.     

Curve registration

Functional data analysis involves the extension of familiar statistical procedures such as principal components analysis, linear modelling, and canonical correlation analysis to data where the raw observation x_i is a function. An essential preliminary to a functional data analysis is often the registration or alignment of salient curve features by suitable monotone transformations h_i of the argument t , so that the actual analyses are carried out on the values x_i { h_i ( t )}. This is referred to as dynamic time warping in the engineering literature. In effect, this conceptualizes variation among functions as being composed of two aspects: horizontal and vertical, or domain and range. A nonparametric function estimation technique is described for identifying the smooth monotone transformations h_i , and is illustrated by data analyses. A second-order linear stochastic differential equation is proposed to model these components of variation.     

SEMI-PARAMETRIC ANALYSIS OF COVARIANCE UNDER DEPENDENCE CONDITIONS WITHIN EACH GROUP

Consider the problem of covariance analysis based on regression models whose regression function is the sum of a linear and a non-parametric component. We propose a parametric and a non-parametric statistical test to compare the effects of the linear and non-parametric components, respectively, on the response variable in   L ≥ 2  groups. Serially correlated errors within each group are allowed. The first (second) test compares the differences between the estimates of the parametric (non-parametric) components of each group by means of a Mahalanobis  ( L ²)  distance. The asymptotic distribution of each statistic under the null hypothesis is obtained. A modest simulation study and an application to a real data set illustrate our methodology.