A new hybrid estimation method for the generalized pareto distribution

ABSTRACT

The generalized Pareto distribution (GPD) is important in the analysis of extreme values, especially in modeling exceedances over thresholds. Most of the existing methods for estimating the scale and shape parameters of the GPD suffer from theoretical and/or computational problems. A new hybrid estimation method is proposed in this article, which minimizes a goodness-of-fit measure and incorporates some useful likelihood information. Compared with the maximum likelihood method and other leading methods, our new hybrid estimation method retains high efficiency, reduces the estimation bias, and is computation friendly.     

Extended generalised Pareto models for tail estimation

The most popular approach in extreme value statistics is the modelling of threshold exceedances using the asymptotically motivated generalised Pareto distribution. This approach involves the selection of a high threshold above which the model fits the data well. Sometimes, few observations of a measurement process might be recorded in applications and so selecting a high quantile of the sample as the threshold leads to almost no exceedances. In this paper we propose extensions of the generalised Pareto distribution that incorporate an additional shape parameter while keeping the tail behaviour unaffected. The inclusion of this parameter offers additional structure for the main body of the distribution, improves the stability of the modified scale, tail index and return level estimates to threshold choice and allows a lower threshold to be selected. We illustrate the benefits of the proposed models with a simulation study and two case studies.     

A diagnostic for selecting the threshold in extreme value analysis

A new approach is suggested for choosing the threshold when fitting the Hill estimator of a tail exponent to extreme value data. Our method is based on an easily computed diagnostic, which in turn is founded directly on the Hill estimator itself, 'symmetrized' to remove the effect of the tail exponent but designed to emphasize biases in estimates of that exponent. The attractions of the method are its accuracy, its simplicity and the generality with which it applies. This generality implies that the technique has somewhat different goals from more conventional approaches, which are designed to accommodate the minor component of a postulated two-component Pareto mixture. Our approach does not rely on the second component being Pareto distributed. Nevertheless, in the conventional setting it performs competitively with recently proposed methods, and in more general cases it achieves optimal rates of convergence. A by-product of our development is a very simple and practicable exponential approximation to the distribution of the Hill estimator under departures from the Pareto distribution.     

Comparisons of methods of estimation for a pareto distribution of the first kind

This paper compares methods of estimation for the parameters of a Pareto distribution of the first kind to determine which method provides the better estimates when the observations are censored, The unweighted least squares (LS) and the maximum likelihood estimates (MLE) are presented for both censored and uncensored data. The MLE's are obtained using two methods, In the first, called the ML method, it is shown that log-likelihood is maximized when the scale parameter is the minimum sample value. In the second method, called the modified ML (MML) method, the estimates are found by utilizing the maximum likelihood value of the shape parameter in terms of the scale parameter and the equation for the mean of the first order statistic as a function of both parameters. Since censored data often occur in applications, we study two types of censoring for their effects on the methods of estimation: Type II censoring and multiple random censoring. In this study we consider different sample sizes and several values of the true shape and scale parameters.

Comparisons are made in terms of bias and the mean squared error of the estimates. We propose that the LS method be generally preferred over the ML and MML methods for estimating the Pareto parameter γ for all sample sizes, all values of the parameter and for both complete and censored samples. In many cases, however, the ML estimates are comparable in their efficiency, so that either estimator can effectively be used. For estimating the parameter α, the LS method is also generally preferred for smaller values of the parameter (α ≤4). For the larger values of the parameter, and for censored samples, the MML method appears superior to the other methods with a slight advantage over the LS method. For larger values of the parameter α, for censored samples and all methods, underestimation can be a problem.     

Extremal memory of stochastic volatility with an application to tail shape inference

We characterize joint tails and tail dependence for a class of stochastic volatility processes. We derive the exact joint tail shape of multivariate stochastic volatility with innovations that have a regularly varying distribution tail. This is used to give four new characterizations of tail dependence. In three cases tail dependence is a non-trivial function of linear volatility memory parametrically represented by tail scales, while tail power indices do not provide any relevant dependence information. Although tail dependence is associated with linear volatility memory, tail dependence itself is nonlinear. In the fourth case a linear function of tail events and exceedances is linearly independent. Tail dependence falls in a class that implies the celebrated Hill (1975) tail index estimator is asymptotically normal, while linear independence of nonlinear tail arrays ensures the asymptotic variance is the same as the iid case. We illustrate the latter finding by simulation.     

The NPMLE for Doubly Censored Current Status Data

In biostatistical applications interest often focuses on the estimation of the distribution of time T between two consecutive events. If the initial event time is observed and the subsequent event time is only known to be larger or smaller than an observed point in time, then the data is described by the well understood singly censored current status model, also known as interval censored data, case I. Jewell et al. (1994) extended this current status model by allowing the initial time to be unobserved, but with its distribution over an observed interval ' A, B ' known to be uniformly distributed; the data is referred to as doubly censored current status data. These authors used this model to handle application in AIDS partner studies focusing on the NPMLE of the distribution G of T . The model is a submodel of the current status model, but the distribution G is essentially the derivative of the distribution of interest F in the current status model. In this paper we establish that the NPMLE of G is uniformly consistent and that the resulting estimators for the n ^1/2-estimable parameters are efficient. We propose an iterative weighted pool-adjacent-violator-algorithm to compute the estimator. It is also shown that, without smoothness assumptions, the NPMLE of F converges at rate n ^−2/5 in L ²-norm while the NPMLE of F in the non-parametric current status data model converges at rate n ^−1/3 in L ²-norm, which shows that there is a substantial gain in using the submodel information.     

A computational algorithm for the loss probability in the M/G/1+PH queue

This article develops a computational algorithm for the loss probability in the stationary M/G/1 queue with impatient customers whose impatience times follow a phase-type distribution (M/G/1+PH). The algorithm outputs the loss probability, along with an upper-bound of its numerical error due to truncation, and it is readily applicable to the M/D/1+PH, M/PH/1+PH, and M/Pareto/1+PH queues.     

On the identification of extreme outliers and dragon-kings mechanisms in the upper tail of income distribution

The presence of extreme outliers in the upper tail data of income distribution affects the Pareto tail modeling. A simulation study is carried out to compare the performance of three types of boxplot in the detection of extreme outliers for Pareto data, including standard boxplot, adjusted boxplot and generalized boxplot. It is found that the generalized boxplot is the best method for determining extreme outliers for Pareto distributed data. For the application, the generalized boxplot is utilized for determining the exreme outliers in the upper tail of Malaysian income distribution. In addition, for this data set, the confidence interval method is applied for examining the presence of dragon-kings, extreme outliers which are beyond the Pareto or power-laws distribution.     

Tests for the Validity of the Assumption that the Underlying Distribution of Life is Pareto

This article considers the problem of testing the validity of the assumption that the underlying distribution of life is Pareto. For complete and censored samples, the relationship between the Pareto and the exponential distributions could be of vital importance to test for the validity of this assumption. For grouped uncensored data the classical Pearson χ² test based on the multinomial model can be used. Attention is confined in this article to handle grouped data with withdrawals within intervals. Graphical as well as analytical procedures will be presented. Maximum likelihood estimators for the parameters of the Pareto distribution based on grouped data will be derived.     

Parameter estimation of the generalized Pareto distribution—Part II

This is the second part of a paper which focuses on reviewing methods for estimating the parameters of the generalized Pareto distribution (GPD). The GPD is a very important distribution in the extreme value context. It is commonly used for modeling the observations that exceed very high thresholds. The ultimate success of the GPD in applications evidently depends on the parameter estimation process. Quite a few methods exist in the literature for estimating the GPD parameters. Estimation procedures, such as the maximum likelihood (ML), the method of moments (MOM) and the probability weighted moments (PWM) method were described in Part I of the paper. We shall continue to review methods for estimating the GPD parameters, in particular methods that are robust and procedures that use the Bayesian methodology. As in Part I, we shall focus on those that are relatively simple and straightforward to be applied to real world data.     

Design-based confidence limits for a distribution function

A method for constructing confidence limits for a distribution function is proposed. This method is a simple modification of the common method based on a normal approximation to the distribution of the estimated distribution function. The methods differ in how the estimated standard errors are used. The coverage properties of the two methods are compared in a simulation study. Coverage probabilities for the proposed method are found to be much closer to the nominal levels, particularly in the tails of the population distribution.     

Computational comparison of the general weighted moments estimators of the scale parameter of a Pareto distribution with a known shape parameter with other estimators based on a multiply type II-censored sample

Wu et al. [Computational comparison for weighted moments estimators and BLUE of the scale parameter of a Pareto distribution with known shape parameter under type II multiply censored sample, Appl. Math. Comput. 181 (2006), pp. 1462–1470] proposed the weighted moments estimators (WMEs) of the scale parameter of a Pareto distribution with known shape parameter on a multiply type II-censored sample. They claimed that some WMEs are better than the best linear unbiased estimator (BLUE) based on the exact mean-squared error (MSE). In this paper, the general WME (GWME) is proposed and the computational comparison of the proposed estimator with the WMEs and BLUE is done on the basis of the exact MSE for given sample sizes and different censoring schemes. As a result, the GWME is performing better than the best estimator among 12 WMEs and BLUE for all cases. Therefore, GWME is recommended for use. At last, one example is given to demonstrate the proposed GWME.     

Tail index estimation in small smaples Simulation results for independent and ARCH-type financial return models

Estimation of the tail index of stationary, fat-tailed return distributions is non-trivial since the well-known Hill estimator is optimal only under iid draws from an exact Pareto model. We provide a small sample simulation study of recently suggested adaptive estimators under ARCH-type dependence. The Hill estimator’s performance is found to be dominated by a ratio estimator. Dependence increases estimation error which can remain substantial even in larger data sets. As small sample bias is related to the magnitude of the tail index, recent standard applications may have overestimated (underestimated) the risk of assets with low (high) degrees of fat-tailedness. This paper is a shortened version of the Berkeley Research Program in Finance Working Paper RPF-295. Thanks are to the Center for Mathematical Sciences at Munich University of Technology for generously providing access to computer facilities and to participants at the IAFE 2001 Budapest, OR 2002 Klagenfurt, EIR 2002 London, DGF 2002 Cologne, FBI 2002 Karlsruhe conferences and the 2001 Wallis Workshop for helpful comments. Two anonymous referees provided helpful suggestions in streamlining the material. Niklas Wagner acknowledges a Maple program by Klaus Kiefersbeck and financial support by Deutsche Forschungsgemeinschaft (DFG).     

Singly and Doubly Censored Current Status Data: Estimation, Asymptotics and Regression

In biostatistical applications interest often focuses on the estimation of the distribution of time between two consecutive events. If the initial event time is observed and the subsequent event time is only known to be larger or smaller than an observed point in time, then the data is described by the well-understood singly censored current status model, also known as interval censored data, case I. Jewell et al. (1994) extended this current status model by allowing the initial time to be unobserved, with its distribution over an observed interval [A, B] known; the data is referred to as doubly censored current status data. This model has applications in AIDS partner studies. If the initial time is known to be uniformly distribute d, the model reduces to a submodel of the current status model with the same asymptotic information bounds as in the current status model, but the distribution of interest is essentially the derivative of the distribution of interest in the current status model. As a consequence the non-parametric maximum likelihood estimator is inconsistent. Moreover, this submodel contains only smooth heavy tailed distributions for which no moments exist. In this paper, we discuss the connection between the singly censored current status model and the doubly censored current status model (for the uniform initial time) in detail and explain the difficulties in estimation which arise in the doubly censored case. We propose a regularized MLE corresponding with the current status model. We prove rate results, efficiency of smooth functionals of the regularized MLE, and present a generally applicable efficient method for estimation of regression parameters, which does not rely on the existence of moments. We also discuss extending these ideas to a non-uniform distribution for the initial time.     

Measuring Tail Thickness to Estimate the Stable Index α: A Critique

A generalized Pareto or simple Pareto tail-index estimate above 2 has frequently been cited as evidence against infinite-variance stable distributions. It is demonstrated that this inference is invalid; tail index estimates greater than 2 are to be expected for stable distributions with α as low as 1.65. The nonregular distribution of the likelihood ratio statistic for a null of normality and an alternative of symmetric stability is tabulated by Monte Carlo methods and appropriately adjusted for sampling error in repeated tests. Real stock returns yield a stable α of 1.845 and reject iid normality at the .996 level.     

A robust prediction error criterion for pareto modelling of upper tails

Estimation of the Pareto tail index from extreme order statistics is an important problem in many settings. The upper tail of the distribution, where data are sparse, is typically fitted with a model, such as the Pareto model, from which quantities such as probabilities associated with extreme events are deduced. The success of this procedure relies heavily not only on the choice of the estimator for the Pareto tail index but also on the procedure used to determine the number k of extreme order statistics that are used for the estimation. The authors develop a robust prediction error criterion for choosing k and estimating the Pareto index. A Monte Carlo study shows the good performance of the new estimator and the analysis of real data sets illustrates that a robust procedure for selection, and not just for estimation, is needed.     

A bias-reduced estimator for the mean of a heavy-tailed distribution with an infinite second moment

We use bias-reduced estimators of high quantiles of heavy-tailed distributions, to introduce a new estimator for the mean in the case of infinite second moment. The asymptotic normality of the proposed estimator is established and checked in a simulation study, by four of the most popular goodness-of-fit tests. The accuracy of the resulting confidence intervals is evaluated as well. We also investigate the finite sample behavior and compare our estimator with some versions of Peng's estimator of the mean (namely those based on Hill, t-Hill and Huisman et al. extreme value index estimators). Moreover, we discuss the robustness of the tail index estimators used in this paper. Finally, our estimation procedure is applied to the well-known Danish fire insurance claims data set, to provide confidence bounds for the means of weekly and monthly maximum losses over a period of 10 years.     

Tail-Adaptive Location Rank Test for the Generalized Secant Hyperbolic Distribution

The generalized secant hyperbolic distribution (GSHD) was recently introduced as a modeling tool in data analysis. The GSHD is a unimodal distribution that is completely specified by location, scale, and shape parameters. It has also been shown elsewhere that the rank procedures of location are regular, robust, and asymptotically fully efficient. In this article, we study certain tail weight measures for the GSHD and introduce a tail-adaptive rank procedure of location based on those tail weight measures. We investigate the properties of the new adaptive rank procedure and compare it to some conventional estimators.     

Irreconcilability of P-value and Bayesian measure in two-sided hypothesis: Pareto distribution with the presence of nuisance parameter

Abstract

This article is concerned with the comparison of Bayesian and classical testing of a point null hypothesis for the Pareto distribution when there is a nuisance parameter. In the first stage, using a fixed prior distribution, the posterior probability is obtained and compared with the P-value. In the second case, lower bounds of the posterior probability of H₀, under a reasonable class of prior distributions, are compared with the P-value. It has been shown that even in the presence of nuisance parameters for the model, these two approaches can lead to different results in statistical inference.     

Parameter estimation for generalized Pareto distribution by generalized probability weighted moment-equations

The generalized Pareto distribution (GPD) has been widely used to model exceedances over a threshold. This article generalizes the method of generalized probability weighted moments, and applies this method to estimate the parameters of GPD. The estimator is computationally easy. Some asymptotic results of this method are provided. Two simulations are carried out to investigate the behavior of this method and to compare them with other methods suggested in the literature. The simulation results show that the performance of the proposed method is better than some other methods. Finally, this method is applied to analyze a real-life data.