Relative smoothing of discrete distributions with sparse observations

Quite often we are faced with a sparse number of observations over a finite number of cells and are interested in estimating the cell probabilities. Some local polynomial smoothers or local likelihood estimators have been proposed to improve on the histogram, which would produce too many zero values. We propose a relativized local polynomial smoothing for this problem, weighting heavier the estimating errors in small probability cells. A simulation study about the estimators that are proposed show a good behaviour with respect to natural error criteria, especially when dealing with sparse observations.     

Recursive and en-bloc approaches to signal extraction

In the literature on unobservable component models , three main statistical instruments have been used for signal extraction: fixed interval smoothing (FIS), which derives from Kalman's seminal work on optimal state-space filter theory in the time domain; Wiener-Kolmogorov-Whittle optimal signal extraction (OSE) theory, which is normally set in the frequency domain and dominates the field of classical statistics; and regularization , which was developed mainly by numerical analysts but is referred to as 'smoothing' in the statistical literature (such as smoothing splines, kernel smoothers and local regression). Although some minor recognition of the interrelationship between these methods can be discerned from the literature, no clear discussion of their equivalence has appeared. This paper exposes clearly the interrelationships between the three methods; highlights important properties of the smoothing filters used in signal extraction; and stresses the advantages of the FIS algorithms as a practical solution to signal extraction and smoothing problems. It also emphasizes the importance of the classical OSE theory as an analytical tool for obtaining a better understanding of the problem of signal extraction.     

A comparison of six smoothers when there are multiple predictors

The paper compares six smoothers, in terms of mean squared error and bias, when there are multiple predictors and the sample size is relatively small. The focus is on methods that use robust measures of location (primarily a 20% trimmed mean) and where there are four predictors. To add perspective, some methods designed for means are included. The smoothers include the locally weighted (loess) method derived by Cleveland and Devlin [W.S. Cleveland, S.J. Devlin, Locally-weighted regression: an approach to regression analysis by local fitting, Journal of the American Statistical Association 83 (1988) 596–610], a variation of a so-called running interval smoother where distances from a point are measured via a particular group of projections of the data, a running interval smoother where distances are measured based in part using the minimum volume ellipsoid estimator, a generalized additive model based on the running interval smoother, a generalized additive model based on the robust version of the smooth derived by Cleveland [W.S. Cleveland, Robust locally weighted regression and smoothing scatterplots, Journal of the American Statistical Association 74 (1979) 829–836], and a kernel regression method stemming from [J. Fan, Local linear smoothers and their minimax efficiencies, The Annals of Statistics 21 (1993) 196–216]. When the regression surface is a plane, the method stemming from [J. Fan, Local linear smoothers and their minimax efficiencies, The Annals of Statistics 21 (1993) 196–216] was found to dominate, and indeed offers a substantial advantage in various situations, even when the error term has a heavy-tailed distribution. However, if there is curvature, this method can perform poorly compared to the other smooths considered. Now the projection-type smoother used in conjunction with a 20% trimmed mean is recommended with the minimum volume ellipsoid method a close second.     

P-splines with derivative based penalties and tensor product smoothing of unevenly distributed data

The P-splines of Eilers and Marx (Stat Sci 11:89–121, 1996) combine a B-spline basis with a discrete quadratic penalty on the basis coefficients, to produce a reduced rank spline like smoother. P-splines have three properties that make them very popular as reduced rank smoothers: (i) the basis and the penalty are sparse, enabling efficient computation, especially for Bayesian stochastic simulation; (ii) it is possible to flexibly ‘mix-and-match’ the order of B-spline basis and penalty, rather than the order of penalty controlling the order of the basis as in spline smoothing; (iii) it is very easy to set up the B-spline basis functions and penalties. The discrete penalties are somewhat less interpretable in terms of function shape than the traditional derivative based spline penalties, but tend towards penalties proportional to traditional spline penalties in the limit of large basis size. However part of the point of P-splines is not to use a large basis size. In addition the spline basis functions arise from solving functional optimization problems involving derivative based penalties, so moving to discrete penalties for smoothing may not always be desirable. The purpose of this note is to point out that the three properties of basis-penalty sparsity, mix-and-match penalization and ease of setup are readily obtainable with B-splines subject to derivative based penalization. The penalty setup typically requires a few lines of code, rather than the two lines typically required for P-splines, but this one off disadvantage seems to be the only one associated with using derivative based penalties. As an example application, it is shown how basis-penalty sparsity enables efficient computation with tensor product smoothers of scattered data.     

Some theoretical results on approximation methods 1

In this paper we give an approximation procedure to surfaces which are defined on a _p-dimensional region and are observable (disturbed with some noice) according to an experimental design. In this procedure we combine clustering methods, discriminant analysis and smoothing techniques.

In the second part of the paper is considered some investigations on statistical properties of linear smoothing procedures. We assume linear models and for a broad class of models we prove the consistence of the estimation of the expectation of observations after smoothing.

In the last section we give some results on efficiency.     

Local regression for vector responses

We explore a class of vector smoothers based on local polynomial regression for fitting nonparametric regression models which have a vector response. The asymptotic bias and variance for the class of estimators are derived for two different ways of representing the variance matrices within both a seemingly unrelated regression and a vector measurement error framework. We show that the asymptotic behaviour of the estimators is different in these four cases. In addition, the placement of the kernel weights in weighted least squares estimators is very important in the seeming unrelated regressions problem (to ensure that the estimator is asymptotically unbiased) but not in the vector measurement error model. It is shown that the component estimators are asymptotically uncorrelated in the seemingly unrelated regressions model but asymptotically correlated in the vector measurement error model. These new and interesting results extend our understanding of the problem of smoothing dependent data.     

Iterative bias reduction: a comparative study

Multivariate nonparametric smoothers, such as kernel based smoothers and thin plate splines smoothers, are adversely impacted by the sparseness of data in high dimension, also known as the curse of dimensionality. Adaptive smoothers, that can exploit the underlying smoothness of the regression function, may partially mitigate this effect. This paper presents a comparative simulation study of a novel adaptive smoother (IBR) with competing multivariate smoothers available as package or function within the R language and environment for statistical computing. Comparison between the methods are made on simulated datasets of moderate size, from 50 to 200 observations, with two, five or 10 potential explanatory variables, and on a real dataset. The results show that the good asymptotic properties of IBR are complemented by a very good behavior on moderate sized datasets, results which are similar to those obtained with Duchon low rank splines.     

Smoothing Time Series with Local Polynomial Regression on Time

Time series smoothers estimate the level of a time series at time t as its conditional expectation given present, past and future observations, with the smoothed value depending on the estimated time series model. Alternatively, local polynomial regressions on time can be used to estimate the level, with the implied smoothed value depending on the weight function and the bandwidth in the local linear least squares fit. In this article we compare the two smoothing approaches and describe their similarities. Through simulations, we assess the increase in the mean square error that results when approximating the estimated optimal time series smoother with the local regression estimate of the level.     

Sparse additive models

Summary.  We present a new class of methods for high dimensional non-parametric regression and classification called sparse additive models. Our methods combine ideas from sparse linear modelling and additive non-parametric regression. We derive an algorithm for fitting the models that is practical and effective even when the number of covariates is larger than the sample size. Sparse additive models are essentially a functional version of the grouped lasso of Yuan and Lin. They are also closely related to the COSSO model of Lin and Zhang but decouple smoothing and sparsity, enabling the use of arbitrary non-parametric smoothers. We give an analysis of the theoretical properties of sparse additive models and present empirical results on synthetic and real data, showing that they can be effective in fitting sparse non-parametric models in high dimensional data.     

Thin plate regression splines

Summary. I discuss the production of low rank smoothers for d  ≥ 1 dimensional data, which can be fitted by regression or penalized regression methods. The smoothers are constructed by a simple transformation and truncation of the basis that arises from the solution of the thin plate spline smoothing problem and are optimal in the sense that the truncation is designed to result in the minimum possible perturbation of the thin plate spline smoothing problem given the dimension of the basis used to construct the smoother. By making use of Lanczos iteration the basis change and truncation are computationally efficient. The smoothers allow the use of approximate thin plate spline models with large data sets, avoid the problems that are associated with 'knot placement' that usually complicate modelling with regression splines or penalized regression splines, provide a sensible way of modelling interaction terms in generalized additive models, provide low rank approximations to generalized smoothing spline models, appropriate for use with large data sets, provide a means for incorporating smooth functions of more than one variable into non-linear models and improve the computational efficiency of penalized likelihood models incorporating thin plate splines. Given that the approach produces spline-like models with a sparse basis, it also provides a natural way of incorporating unpenalized spline-like terms in linear and generalized linear models, and these can be treated just like any other model terms from the point of view of model selection, inference and diagnostics.     

CoSmo: A Constrained Scatterplot Smoother for Estimating Convex,Monotonic Transformations

In many of the applied sciences, it is common that the forms of empirical relationships are almost completely unknown prior to study. Scatterplot smoothers used in nonparametric regression methods have considerable potential to ease the burden of model specification that a researcher would otherwise face in this situation. Occasionally the researcher will know the sign of the first or second derivatives, or both. This article develops a smoothing method that can incorporate this kind of information. I show that cubic regression splines with bounds on the coefficients offer a simple and effective approximation to monotonic, convex or concave transformations. I also discuss methods for testing whether the constraints should be imposed. Monte Carlo results indicate that this method, dubbed CoSmo, has a lower approximation error than either locally weighted regression or two other constrained smoothing methods. CoSmo has many potential applications and should be especially useful in applied econometrics. As an illustration, I apply CoSmo in a multivariate context to estimate a hedonic price function and to test for concavity in one of the variables.     

Diagnostic tools in generalized Weibull linear regression models

We propose some statistical tools for diagnosing the class of generalized Weibull linear regression models [A.A. Prudente and G.M. Cordeiro, Generalized Weibull linear models, Comm. Statist. Theory Methods 39 (2010), pp. 3739–3755]. This class of models is an alternative means of analysing positive, continuous and skewed data and, due to its statistical properties, is very competitive with gamma regression models. First, we show that the Weibull model induces ma-ximum likelihood estimators asymptotically more efficient than the gamma model. Standardized residuals are defined, and their statistical properties are examined empirically. Some measures are derived based on the case-deletion model, including the generalized Cook's distance and measures for identifying influential observations on partial F-tests. The results of a simulation study conducted to assess behaviour of the global influence approach are also presented. Further, we perform a local influence analysis under the case-weights, response and explanatory variables perturbation schemes. The Weibull, gamma and other Weibull-type regression models are fitted into three data sets to illustrate the proposed diagnostic tools. Statistical analyses indicate that the Weibull model fitted into these data yields better fits than other common alternative models.     

Estimation of the error distribution in a varying coefficient regression model

This paper deals with the estimation of the error distribution function in a varying coefficient regression model. We propose two estimators and study their asymptotic properties by obtaining uniform stochastic expansions. The first estimator is a residual-based empirical distribution function. We study this estimator when the varying coefficients are estimated by under-smoothed local quadratic smoothers. Our second estimator which exploits the fact that the error distribution has mean zero is a weighted residual-based empirical distribution whose weights are chosen to achieve the mean zero property using empirical likelihood methods. The second estimator improves on the first estimator. Bootstrap confidence bands based on the two estimators are also discussed.     

On cross-validation for discrete kernel estimates in discrimination

The choice of smoothing determines the properties of nonparametric estimates of probability densities. In the discrimination problem, the choice is often tied to loss functions. A framework for the cross–validatory choice of smoothing parameters based on general loss functions is given. Several loss functions are considered as special cases. In particular, a family of loss functions, which is connected to discrimination problems, is directly related to measures of performance used in discrimination. Consistency results are given for a general class of loss functions which comprise this family of discriminant loss functions.     

Minimax Linear Smoothers

We consider the problem of estimating the mean of a multivariate distribution. As a general alternative to penalized least squares estimators, we consider minimax estimators for squared error over a restricted parameter space where the restriction is determined by the penalization term. For a quadratic penalty term, the minimax estimator among linear estimators can be found explicitly. It is shown that all symmetric linear smoothers with eigenvalues in the unit interval can be characterized as minimax linear estimators over a certain parameter space where the bias is bounded. The minimax linear estimator depends on smoothing parameters that must be estimated in practice. Using results in Kneip (1994), this can be done using Mallows' C _L -statistic and the resulting adaptive estimator is now asymptotically minimax linear. The minimax estimator is compared to the penalized least squares estimator both in finite samples and asymptotically.     

Considerations for optimal nonparametric regression under a generalizederror structure

Nonparametric smoothing, such as kernel or spline estimation, has been examined extensively under the assumption of uncorrelated errors. This paper addresses the effects of potential correlation on consistency and other asymptotic properties in a repeated-measures model, using directly optimized linear smoothers of the replicate means. Unrestricted optimal weights, with respect to squared error loss, are used to confirm a lack of consistency for all linear estimators in an autocorrelated errors model. The results indicate kernel methods that work well for an uncorrelated errors model may not have the ability to perform satisfactorily when correlation is introduced, due to an asymmetry in the optimal weights, which disappears for an uncorrelated errors model. These would include data-driven bandwidth selection methods, adjustments of the bandwidth to accommodate correlation, higher-order kernels, and related bias reduction techniques. The analytic results suggest alternative approaches, not considered here in detail, which have shown merit.     

On the Uniform Strong Consistency of Local Polynomial Regression Under Dependence Conditions

Abstract

In this article, nonparametric estimators of the regression function, and its derivatives, obtained by means of weighted local polynomial fitting are studied. Consider the fixed regression model where the error random variables are coming from a stationary stochastic process satisfying a mixing condition. Uniform strong consistency, along with rates, are established for these estimators. Furthermore, when the errors follow an AR(1) correlation structure, strong consistency properties are also derived for a modified version of the local polynomial estimators proposed by Vilar-Fernández and Francisco-Fernández (Vilar-Fernández, J. M., Francisco-Fernández, M. (2002 Vilar-Fernández, J. M. and Francisco-Fernández, M. 2002. Local polynomial regression smoothers with AR-error structure. TEST, 11(2): 439–464.  [Google Scholar]). Local polynomial regression smoothers with AR-error structure. TEST 11(2):439–464).     

Nonparametric particle filtering and smoothing with quasi-Monte Carlo sampling

Sequential Monte Carlo methods (also known as particle filters and smoothers) are used for filtering and smoothing in general state-space models. These methods are based on importance sampling. In practice, it is often difficult to find a suitable proposal which allows effective importance sampling. This article develops an original particle filter and an original particle smoother which employ nonparametric importance sampling. The basic idea is to use a nonparametric estimate of the marginally optimal proposal. The proposed algorithms provide a better approximation of the filtering and smoothing distributions than standard methods. The methods’ advantage is most distinct in severely nonlinear situations. In contrast to most existing methods, they allow the use of quasi-Monte Carlo (QMC) sampling. In addition, they do not suffer from weight degeneration rendering a resampling step unnecessary. For the estimation of model parameters, an efficient on-line maximum-likelihood (ML) estimation technique is proposed which is also based on nonparametric approximations. All suggested algorithms have almost linear complexity for low-dimensional state-spaces. This is an advantage over standard smoothing and ML procedures. Particularly, all existing sequential Monte Carlo methods that incorporate QMC sampling have quadratic complexity. As an application, stochastic volatility estimation for high-frequency financial data is considered, which is of great importance in practice. The computer code is partly available as supplemental material.     

Linear sensitivity measures in statistical disclosure control

The mathematical properties of a class of functions called linear sensitivity measures are investigated. These measures are applied to the problem of maintaining the statistical confidentiality of respondents to a census or statistical survey such as an establishment-based economic survey. Sensitivity criteria in practical use are cast in this setting.     

Nonparametric monitoring of financial time series by jump-preserving control charts

Since structural changes in a possibly transformed financial time series may contain important information for investors and analysts, we consider the following problem of sequential econometrics. For a given time series we aim at detecting the first change-point where a jump of size a occurs, i.e., the mean changes from, say, m ₀to m ₀+ a and returns to m ₀after a possibly short period s. To address this problem, we study a Shewhart-type control chart based on a sequential version of the sigma filter, which extends kernel smoothers by employing stochastic weights depending on the process history to detect jumps in the data more accurately than classical approaches. We study both theoretical properties and performance issues. Concerning the statistical properties, it is important to know whether the normed delay time of the considered control chart is bounded, at least asymptotically. Extending known results for linear statistics employing deterministic weighting schemes, we establish an upper bound which holds if the memory of the chart tends to infinity. The performance of the proposed control charts is studied by simulations. We confine ourselves to some special models which try to mimic important features of real time series. Our empirical results provide some evidence that jump-preserving weights are preferable under certain circumstances.