Asymptotical improvement of maximum likelihood estimators on Kullback–Leibler loss

We discuss the general form of a first-order correction to the maximum likelihood estimator which is expressed in terms of the gradient of a function, which could for example be the logarithm of a prior density function. In terms of Kullback–Leibler divergence, the correction gives an asymptotic improvement over maximum likelihood under rather general conditions. The theory is illustrated for Bayes estimators with conjugate priors. The optimal choice of hyper-parameter to improve the maximum likelihood estimator is discussed. The results based on Kullback–Leibler risk are extended to a wide class of risk functions.     

General cumulative Kullback–Leibler information

The cumulative residual Kullback–Leibler information is defined on the semi-infinite (non negative) interval. In this paper, we extend the cumulative residual Kullback–Leibler information to the whole real line and propose a general cumulative Kullback–Leibler information. We study its application to a test for normality in comparison with some competing test statistics based on the empirical distribution function including the well-known tests applied in practice like Kolmogorov–Smirnov, Cramer–von Mises, Anderson–Darling, and other existing tests.     

Quantile-based cumulative Kullback–Leibler divergence

The paper introduces a quantile-based cumulative Kullback–Leibler divergence and study its various properties. Unlike the distribution function approach, the quantile-based measure possesses some unique properties. The quantile functions used in many applied works do not have any tractable distribution functions where the proposed measure is a useful tool to compute the distance between two random variables. Some useful bounds are obtained for quantile-based residual cumulative Kullback–Leibler divergence and quantile-based reliability measures. Characterization results based on the functional forms of quantile-based residual Kullback–Leibler divergence are obtained for some well-known life distributions, namely exponential, Pareto II and beta.     

Kullback–Leibler information of a censored variable and its applications

In this paper, we study the Kullback–Leibler (KL) information of a censored variable, which we will simply call it censored KL information. The censored KL information is shown to have the necessary monotonicity property in addition to inherent properties of nonnegativity and characterization. We also present a representation of the censored KL information in terms of the relative risk and study its relation with the Fisher information in censored data. Finally, we evaluate the estimated censored KL information as a goodness-of-fit test statistic.     

On the Kullback–Leibler information of hybrid censored data

ABSTRACT

A hybrid censoring is a mixture of Type I and Type II censoring where the experiment terminates when either rth failure or predetermined censoring time comes first or later. In this article, we consider order statistics of the Type I censored data and provide a simple expression for their Kullback–Leibler (KL) information. Then, we provide the expressions for the KL information of the Type I and Type II hybrid censored data.     

Optimal Latin hypercube designs for the Kullback–Leibler criterion

Space-filling designs are commonly used for selecting the input values of time-consuming computer codes. Computer experiment context implies two constraints on the design. First, the design points should be evenly spread throughout the experimental region. A space-filling criterion (for instance, the maximin distance) is used to build optimal designs. Second, the design should avoid replication when projecting the points onto a subset of input variables (non-collapsing). The Latin hypercube structure is often enforced to ensure good projective properties. In this paper, a space-filling criterion based on the Kullback–Leibler information is used to build a new class of Latin hypercube designs. The new designs are compared with several traditional optimal Latin hypercube designs and appear to perform well.     

Bayesian analysis of hypothesis testing problems for general population: A Kullback–Leibler alternative

We consider a hypothesis problem with directional alternatives. We approach the problem from a Bayesian decision theoretic point of view and consider a situation when one side of the alternatives is more important or more probable than the other. We develop a general Bayesian framework by specifying a mixture prior structure and a loss function related to the Kullback–Leibler divergence. This Bayesian decision method is applied to Normal and Poisson populations. Simulations are performed to compare the performance of the proposed method with that of a method based on a classical z-test and a Bayesian method based on the “0–1” loss.     

General treatment of goodness-of-fit tests based on Kullback–Leibler information

This paper introduces a general goodness-of-fit test based on the estimated Kullback–Leibler information. The test uses the Vasicek entropy estimate. Two special cases of the test for location–scale and shape families are discussed. The results are used to introduce goodness-of-fit tests for the uniform, Laplace, Weibull and beta distributions. The critical values and powers for some alternatives are obtained by simulation.     

A Goodness-of-Fit Test for the Gumbel Distribution Based on Kullback–Leibler Information

A goodness-of-fit test for the Gumbel distribution is proposed. This test is based on the Kullback–Leibler discrimination information methodology proposed by Song (2002 Song , K. S. ( 2002 ). Goodness-of-fit tests based on Kullback–Leibler discrimination information . IEEE Trans. Inform. Theor. 48 : 1103 – 1117 .[Crossref], [Web of Science ®] , [Google Scholar]). The critical values of the test were obtained by using Monte Carlo simulation for small sample sizes and different levels of significance. The proposed test is compared with the tests developed by Stephens (1977 Stephens , M. ( 1977 ). Goodness-of-fit tests for the extreme value distribution . Biometrika 65 : 730 – 737 . [Google Scholar]), Chandra et al. (1981 Chandra , M. , Singpurwalla , N. D. , Stephens , M. A. ( 1981 ). Kolmogorov statistics for tests of fit for the extreme value and Weibull distributions . J. Amer. Statist. Assoc. 74 : 729 – 735 . [Google Scholar]), and the test given by Kinnison (1989 Kinnison , R. (1989). Correlation coefficient goodness of fit test for the extreme value distribution. Amer. Statistician 43:98–100.[Taylor & Francis Online], [Web of Science ®] , [Google Scholar]) in terms of their power by considering various alternative distributions. Simulation results show that the Kullback–Leibler information test has higher power than some of the studied tests.     

Goodness-of-fit tests based on Verma Kullback–Leibler information

In this article, a new consistent estimator of Veram’s entropy is introduced. We establish the entropy test based on the new information namely Verma Kullback–Leibler discrimination methodology. The results are used to introduce goodness-of-fit tests for normal and exponential distributions. The root of mean square errors, critical values, and powers for some alternatives are obtained by simulation. The proposed test is compared with other tests.     

A new estimator of Kullback–Leibler information and its application in goodness of fit tests*

We propose here a general statistic for the goodness of fit test of statistical distributions. The proposed statistic is constructed based on an estimate of Kullback–Leibler information. The proposed test is consistent and the limiting distribution of the test statistic is derived. Then, the established results are used to introduce goodness of fit tests for the normal, exponential, Laplace and Weibull distributions. A simulation study is carried out for examining the power of the proposed test and to compare it with those of some existing procedures. Finally, some illustrative examples are presented and analysed, and concluding comments are made.     

A note on consistent estimation of Kullback–Leibler discrepancy in Poisson regression

A recent theorem by Hannig and Lee on consistency of their estimator of Kullback–Leibler discrepancy is re-proved under assumptions suitably modified to correct a fault in the original proof.     

Characterizing variation of nonparametric random probability measures using the Kullback–Leibler divergence

This work characterizes the dispersion of some popular random probability measures, including the bootstrap, the Bayesian bootstrap, and the Pólya tree prior. This dispersion is measured in terms of the variation of the Kullback–Leibler divergence of a random draw from the process to that of its baseline centring measure. By providing a quantitative expression of this dispersion around the baseline distribution, our work provides insight for comparing different parameterizations of the models and for the setting of prior parameters in applied Bayesian settings. This highlights some limitations of the existing canonical choice of parameter settings in the Pólya tree process.     

Alternatives to maximum likelihood estimation based on spacings and the Kullback–Leibler divergence

An alternative to the maximum likelihood (ML) method, the maximum spacing (MSP) method, is introduced in Cheng and Amin [1983. Estimating parameters in continuous univariate distributions with a shifted origin. J. Roy. Statist. Soc. Ser. B 45, 394–403], and independently in Ranneby [1984. The maximum spacing method. An estimation method related to the maximum likelihood method. Scand. J. Statist. 11, 93–112]. The method, as described by Ranneby [1984. The maximum spacing method. An estimation method related to the maximum likelihood method. Scand. J. Statist. 11, 93–112], is derived from an approximation of the Kullback–Leibler divergence. Since the introduction of the MSP method, several closely related methods have been suggested. This article is a survey of such methods based on spacings and the Kullback–Leibler divergence. These estimation methods possess good properties and they work in situations where the ML method does not. Important issues such as the handling of ties and incomplete data are discussed, and it is argued that by using Moran's [1951. The random division of an interval—Part II. J. Roy. Statist. Soc. Ser. B 13, 147–150] statistic, on which the MSP method is based, we can effectively combine: (a) a test on whether an assigned model of distribution functions is correct or not, (b) an asymptotically efficient estimation of an unknown parameter _θ0${\theta }_0$, and (c) a computation of a confidence region for _θ0${\theta }_0$.     

Testing exponentiality based on Kullback—Leibler information for progressively Type II censored data

In many life-testing and reliability experiments, data are often censored in order to reduce the cost and time associated with testing and since the conventional Type-I and Type-II censoring schemes are not flexible enough, progressive censoring is developed by researchers. In this article, we develop a general goodness of fit test by using a new estimate of Kullback–Leibler information based on progressively Type-II censored data. Consistency and other properties of the proposed test are shown. Then, we use the proposed test statistic to test for exponentiality based on progressively Type-II censored data. The power values of the proposed test under different progressively Type-II censoring schemes are computed, through Monte Carlo simulations. It is observed that the proposed test is quite powerful in compared with the test proposed by Balakrishnan et al. (2007 Balakrishnan, N., Habibi Rad, A., and Arghami, N. R. (2007). Testing exponentiality based on Kullback–Leibler information with progressively type-II censored data. IEEE Transactions on Reliability 56:301–307. [Google Scholar]). Two real datasets from progressive censoring literature are finally presented for illustrative purpose.     

Marcinkiewicz–Zygmund and ordinary strong laws for empirical distribution functions and plug-in estimators

Both Marcinkiewicz–Zygmund strong laws of large numbers (MZ-SLLNs) and ordinary strong laws of large numbers (SLLNs) for plug-in estimators of general statistical functionals are derived. It is used that if a statistical functional is ‘sufficiently regular’, then an (MZ-)SLLN for the estimator of the unknown distribution function yields an (MZ-)SLLN for the corresponding plug-in estimator. It is in particular shown that many L-, V- and risk functionals are ‘sufficiently regular’ and that known results on the strong convergence of the empirical process of α-mixing random variables can be improved. The presented approach does not only cover some known results but also provides some new strong laws for plug-in estimators of particular statistical functionals.     

Admissible minimax estimators for the shape parameter of Topp–Leone distribution

Abstract

The shape parameter of Topp–Leone distribution is estimated in this article from the Bayesian viewpoint under the assumption of known scale parameter. Bayes and empirical Bayes estimates of the unknown parameter are proposed under non informative and suitable conjugate priors. These estimates are derived under the assumption of squared and linear-exponential error loss functions. The risk functions of the proposed estimates are derived in analytical forms. It is shown that the proposed estimates are minimax and admissible. The consistency of the proposed estimates under the squared error loss function is also proved. Numerical examples are provided.