Smoothing Splines and Shape Restrictions

Constrained smoothing splines are discussed under order restrictions on the shape of the function m . We consider shape constraints of the type m ^{( r )}≥ 0, i.e. positivity, monotonicity, convexity, .... (Here for an integer r ≥ 0, m ^{( r )} denotes the r th derivative of m .) The paper contains three results: (1) constrained smoothing splines achieve optimal rates in shape restricted Sobolev classes; (2) they are equivalent to two step procedures of the following type: (a) in a first step the unconstrained smoothing spline is calculated; (b) in a second step the unconstrained smoothing spline is "projected" onto the constrained set. The projection is calculated with respect to a Sobolev-type norm; this result can be used for two purposes, it may motivate new algorithmic approaches and it helps to understand the form of the estimator and its asymptotic properties; (3) the infinite number of constraints can be replaced by a finite number with only a small loss of accuracy, this is discussed for estimation of a convex function.     

Shape constrained kernel density estimation

In this paper, a method for estimating monotone, convex and log-concave densities is proposed. The estimation procedure consists of an unconstrained kernel estimator which is modified in a second step with respect to the desired shape constraint by using monotone rearrangements. It is shown that the resulting estimate is a density itself and shares the asymptotic properties of the unconstrained estimate. A short simulation study shows the finite sample behavior.     

Strictly monotone and smooth nonparametric regression for two or more variables

The authors propose a new monotone nonparametric estimate for a regression function of two or more variables. Their method consists in applying successively one‐dimensional isotonization procedures on an initial, unconstrained nonparametric regression estimate. In the case of a strictly monotone regression function, they show that the new estimate and the initial one are first‐order asymptotic equivalent; they also establish asymptotic normality of an appropriate standardization of the new estimate. In addition, they show that if the regression function is not monotone in one of its arguments, the new estimate and the initial one have approximately the same L^p‐norm. They illustrate their approach by means of a simulation study, and two data examples are analyzed.     

Weighted least-squares rank estimates

In this paper rank estimates, called WLS rank estimates and computed using iteratively reweighted least squares, are studied. They do not require the estimation of auxilliary scale or slope parameters nor do they require numerical search techniques to minimize a convex surface. The price is a small asymptotic efficiency loss. In the location model, beginning with a resistant starting value such as the median, the WLS rank estimates have good robustness and computational properties. The WLS rank estimate is also extended to the regression model and an example is given.     

Testing the monotonicity or convexity of a function using regression splines

Methods for smoothed isotonic or convex regression are useful in many applications. Sometimes the shape assumptions constitute a priori knowledge about the regression function, but often the shape is part of the research question. The authors propose tests for monotonicity and convexity using constrained and unconstrained regression splines. The tests have good large‐sample properties and the small‐sample behaviour is illustrated through simulations. Extensions to the partial linear model and the generalized regression model are presented. The Canadian Journal of Statistics 39: 89–107; 2011 © 2011 Statistical Society of Canada     

An Algorithm for Unconstrained Quadratically Penalized Convex Optimization

Estimators are often defined as the solutions to data dependent optimization problems. A common form of objective function (function to be optimized) that arises in statistical estimation is the sum of a convex function V and a quadratic complexity penalty. A standard paradigm for creating kernel-based estimators leads to such an optimization problem. This article describes an optimization algorithm designed for unconstrained optimization problems in which the objective function is the sum of a non negative convex function and a known quadratic penalty. The algorithm is described and compared with BFGS on some penalized logistic regression and penalized L ^3/2 regression problems.     

Fast Censored Linear Regression

Weighted log‐rank estimating function has become a standard estimation method for the censored linear regression model, or the accelerated failure time model. Well established statistically, the estimator defined as a consistent root has, however, rather poor computational properties because the estimating function is neither continuous nor, in general, monotone. We propose a computationally efficient estimator through an asymptotics‐guided Newton algorithm, in which censored quantile regression methods are tailored to yield an initial consistent estimate and a consistent derivative estimate of the limiting estimating function. We also develop fast interval estimation with a new proposal for sandwich variance estimation. The proposed estimator is asymptotically equivalent to the consistent root estimator and barely distinguishable in samples of practical size. However, computation time is typically reduced by two to three orders of magnitude for point estimation alone. Illustrations with clinical applications are provided.     

Efficient Augmented Inverse Probability Weighted Estimation in Missing Data Problems

When analyzing data with missing data, a commonly used method is the inverse probability weighting (IPW) method, which reweights estimating equations with propensity scores. The popularity of the IPW method is due to its simplicity. However, it is often being criticized for being inefficient because most of the information from the incomplete observations is not used. Alternatively, the regression method is known to be efficient but is nonrobust to the misspecification of the regression function. In this article, we propose a novel way of optimally combining the propensity score function and the regression model. The resulting estimating equation enjoys the properties of robustness against misspecification of either the propensity score or the regression function, as well as being locally semiparametric efficient. We demonstrate analytically situations where our method leads to a more efficient estimator than some of its competitors. In a simulation study, we show the new method compares favorably with its competitors in finite samples. Supplementary materials for this article are available online.     

Smoothing Spline Estimation for Partially Linear Single-index Models

In this article, the partially linear single-index models are discussed based on smoothing spline and average derivative estimation method. This proposed technique consists of two stages: one is to estimate the vector parameter in the linear part using the smoothing cubic spline method, simultaneously, obtaining the estimator of unknown single-index function; the other is to estimate the single-index coefficients in the single-index part by the using average derivative estimator procedure. Some simulated and real examples are presented to illustrate the performance of this method.     

A Bayesian approach to non-parametric monotone function estimation

Summary.  The paper proposes two Bayesian approaches to non-parametric monotone function estimation. The first approach uses a hierarchical Bayes framework and a characterization of smooth monotone functions given by Ramsay that allows unconstrained estimation. The second approach uses a Bayesian regression spline model of Smith and Kohn with a mixture distribution of constrained normal distributions as the prior for the regression coefficients to ensure the monotonicity of the resulting function estimate. The small sample properties of the two function estimators across a range of functions are provided via simulation and compared with existing methods. Asymptotic results are also given that show that Bayesian methods provide consistent function estimators for a large class of smooth functions. An example is provided involving economic demand functions that illustrates the application of the constrained regression spline estimator in the context of a multiple-regression model where two functions are constrained to be monotone.     

Statistical Inferences Based on Extended Weighted Log-Rank Estimating Function for Censored Regression Model

In the regression model with censored data, it is not straightforward to estimate the covariances of the regression estimators, since their asymptotic covariances may involve the unknown error density function and its derivative. In this article, a resampling method for making inferences on the parameter, based on some estimating functions, is discussed for the censored regression model. The inference procedures are associated with a weight function. To find the best weight functions for the proposed procedures, extensive simulations are performed. The validity of the approximation to the distribution of the estimator by a resampling technique is also examined visually. Implementation of the procedures is discussed and illustrated in a real data example.     

Kernel Density-Based Linear Regression Estimate

For linear regression models with non normally distributed errors, the least squares estimate (LSE) will lose some efficiency compared to the maximum likelihood estimate (MLE). In this article, we propose a kernel density-based regression estimate (KDRE) that is adaptive to the unknown error distribution. The key idea is to approximate the likelihood function by using a nonparametric kernel density estimate of the error density based on some initial parameter estimate. The proposed estimate is shown to be asymptotically as efficient as the oracle MLE which assumes the error density were known. In addition, we propose an EM type algorithm to maximize the estimated likelihood function and show that the KDRE can be considered as an iterated weighted least squares estimate, which provides us some insights on the adaptiveness of KDRE to the unknown error distribution. Our Monte Carlo simulation studies show that, while comparable to the traditional LSE for normal errors, the proposed estimation procedure can have substantial efficiency gain for non normal errors. Moreover, the efficiency gain can be achieved even for a small sample size.     

Semiparametric Regression with Kernel Error Model

Abstract.  We propose and study a class of regression models, in which the mean function is specified parametrically as in the existing regression methods, but the residual distribution is modelled non-parametrically by a kernel estimator, without imposing any assumption on its distribution. This specification is different from the existing semiparametric regression models. The asymptotic properties of such likelihood and the maximum likelihood estimate (MLE) under this semiparametric model are studied. We show that under some regularity conditions, the MLE under this model is consistent (when compared with the possibly pseudo-consistency of the parameter estimation under the existing parametric regression model), is asymptotically normal with rate and efficient. The non-parametric pseudo-likelihood ratio has the Wilks property as the true likelihood ratio does. Simulated examples are presented to evaluate the accuracy of the proposed semiparametric MLE method.     

Estimation of exponential regression parameters using binary data

Exponential regression model is important in analyzing data from heterogeneous populations. In this paper we propose a simple method to estimate the regression parameters using binary data. Under certain design distributions, including ellipticaily symmetric distributions, for the explanatory variables, the estimators are shown to be consistent and asymptotically normal when sample size is large. For finite samples, the new estimates were shown to behave reasonably well. They are competitive with the maximum likelihood estimates and more importantly, according to our simulation results, the cost of CPU time for computing new estimates is only 1/7 of that required for computing the usual maximum likelihood estimates. We expect the savings in CPU time would be more dramatic with larger dimension of the regression parameter space.     

Variable selection in quantile regression when the models have autoregressive errors

This paper considers a problem of variable selection in quantile regression with autoregressive errors. Recently, Wu and Liu (2009) investigated the oracle properties of the SCAD and adaptive-LASSO penalized quantile regressions under non identical but independent error assumption. We further relax the error assumptions so that the regression model can hold autoregressive errors, and then investigate theoretical properties for our proposed penalized quantile estimators under the relaxed assumption. Optimizing the objective function is often challenging because both quantile loss and penalty functions may be non-differentiable and/or non-concave. We adopt the concept of pseudo data by Oh et al. (2007) to implement a practical algorithm for the quantile estimate. In addition, we discuss the convergence property of the proposed algorithm. The performance of the proposed method is compared with those of the majorization-minimization algorithm (Hunter and Li, 2005) and the difference convex algorithm (Wu and Liu, 2009) through numerical and real examples.     

Maximum likelihood computation for fitting semiparametric mixture models

Three general algorithms that use different strategies are proposed for computing the maximum likelihood estimate of a semiparametric mixture model. They seek to maximize the likelihood function by, respectively, alternating the parameters, profiling the likelihood and modifying the support set. All three algorithms make a direct use of the recently proposed fast and stable constrained Newton method for computing the nonparametric maximum likelihood of a mixing distribution and employ additionally an optimization algorithm for unconstrained problems. The performance of the algorithms is numerically investigated and compared for solving the Neyman-Scott problem, overcoming overdispersion in logistic regression models and fitting two-level mixed effects logistic regression models. Satisfactory results have been obtained.     

Properties of maximum likelihood estimates of the ratio of parameters in ordinal response regression models

The properties of a method of estimating the ratio of parameters for ordered categorical response regression models are discussed. If the link function relating the response variable to the linear combination of covariates is unknown then it is only possible to estimate the ratio of regression parameters. This ratio of parameters has a substitutability or relative importance interpretation.

The maximum likelihood estimate of the ratio of parameters, assuming a logistic function (McCullagh, 1980), is found to have very small bias for a wide variety of true link functions. Further it is shown using Monte Carlo simulations that this maximum likelihood estimate, has good coverage properties, even if the link function is incorrectly specified. It is demonstrated that combining adjacent categories to make the response binary can result in an analysis which is appreciably less efficient. The size of the efficiency loss on, among other factors, the marginal distribution in the ordered categories     

Testing Monotonicity of Regression Functions – An Empirical Process Approach

We propose several new tests for monotonicity of regression functions based on different empirical processes of residuals and pseudo‐residuals. The residuals are obtained from an unconstrained kernel regression estimator whereas the pseudo‐residuals are obtained from an increasing regression estimator. Here, in particular, we consider a recently developed simple kernel‐based estimator for increasing regression functions based on increasing rearrangements of unconstrained non‐parametric estimators. The test statistics are estimated distance measures between the regression function and its increasing rearrangement. We discuss the asymptotic distributions, consistency and small sample performances of the tests.     

On estimation of the mean parameter of a truncated normal disiribution with known coefficient of variation

This paper eals with the proplem on estimating the mean paramerer of a truncated normal distribution with known coefficient of variation. In the previous treatment of this problem most authors have used the sample standared deviation for estimating this parameter. In the present paper we use Gini’s coefficient of mean difference g and obtain the minimum variance unbiased estimate of the mean based on a linear function of the sample mean and g, It is shown that this new estimate has desirable properties for small samples as well as for large samples. We also give a numerical example.     

Local quadratic estimation of the curvature in a functional single index model

The nonlinear responses of species to environmental variability can play an important role in the maintenance of ecological diversity. Nonetheless, many models use parametric nonlinear terms which pre-determine the ecological conclusions. Motivated by this concern, we study the estimate of the second derivative (curvature) of the link function in a functional single index model. Since the coefficient function and the link function are both unknown, the estimate is expressed as a nested optimization. We first estimate the coefficient function by minimizing squared error where the link function is estimated with a Nadaraya-Watson estimator for each candidate coefficient function. The first and second derivatives of the link function are then estimated via local-quadratic regression using the estimated coefficient function. In this paper, we derive a convergence rate for the curvature of the nonlinear response. In addition, we prove that the argument of the linear predictor can be estimated root-n consistently. However, practical implementation of the method requires solving a nonlinear optimization problem, and our results show that the estimates of the link function and the coefficient function are quite sensitive to the choices of starting values.