Studies of information quantities and information geometry of higher order cumulant spaces

Given a vector-valued stationary time series, we define information quantities such as entropy, divergence, and mutual information by their second- and higher-order cumulant spectra. These quantities are naturally introduced from the information geometrical viewpoint. We present their expressions for linear processes and for random vectors. In the case of linear processes, relations to the identification of transfer function matrices are clarified. In the case of random vectors, relations to the quantities defined by using probability density functions are provided. As an application, we treat the identification of nonlinear systems in the framework of this paper. We also present differential geometrical backgrounds based on the invariance for our definitions.     

Approximation and information topologies

In this paper approximation properties of finite dimensional parametric models are described in terms of an information metric: the Hellinger distance. Under conditions on the parametric family given solely in terms of a comparison of the Hellinger distance with the parameter metric, optimal rates of convergence are described. It is also shown how to use these conditions on the parametric family to determine whether consistent estimation is possible. We give applications of the theorems to regular and non-regular parametric families, and to nonlinear regression.     

Bootstrapping with auxiliary information

In the nonparametric setting, the standard bootstrap method is based on the empirical distribution function of a random sample. The author proposes, by means of the empirical likelihood technique, an alternative bootstrap procedure under a nonparametric model in which one has some auxiliary information about the population distribution. By proving the almost sure weak convergence of the modified bootstrapped empirical process, the validity of the proposed bootstrap procedure is established. This new result is used to obtain bootstrap confidence bands for the population distribution function and to perform the bootstrap Kolmogorov test in the presence of auxiliary information. Other applications include bootstrapping means and variances with auxiliary information. Three simulation studies are presented to demonstrate the performance of the proposed bootstrap procedure for small samples.     

Markov sampling with auxiliary information

In this paper a new class of sampling design, with a markovian behaviour, which use both auxiliary and label information for estimating the total of a finite population, is proposed. The performance of these schemes is numerically compared with Midzuno, Rejective and PPS systematic sampling schemes for populations exhibiting exponential trend.     

An information improvement generating function

Here we define an information improvement generating function whose derivative at point 1 gives Theil's measure of information improvement which has wide applications in Economics. It contains Guiasu and Reischeir's relative information generating function and Golomb's information generating function as particular cases. Simple expressions for important discrete distributions have been obtained. It has also been shown that the information improvement generating function suggests a new information indicator as the standard deviation of the variation of information.     

Marginal information for structure learning

Structure learning for Bayesian networks has been made in a heuristic mode in search of an optimal model to avoid an explosive computational burden. In the learning process, a structural error which occurred at a point of learning may deteriorate its subsequent learning. We proposed a remedial approach to this error-for-error process by using marginal model structures. The remedy is made by fixing local errors in structure in reference to the marginal structures. In this sense, we call the remedy a marginally corrective procedure. We devised a new score function for the procedure which consists of two components, the likelihood function of a model and a discrepancy measure in marginal structures. The proposed method compares favourably with a couple of the most popular algorithms as shown in experiments with benchmark data sets.     

Testing independence with additional information

Let X = (X^j : j = 1,…, n) be n row vectors of dimension p independently and identically distributed multinomial. For each j, X^j is partitioned as X^j = (X^j₁, X^j₂, X^j₃), where p_i is the dimension of X^j_i with p₁ = 1,p₁+p₂+p₃ = p. In addition, consider vectors Y^j_i, i = 1,2j = 1,…,n_i that are independent and distributed as X¹_i. We treat here the problem of testing independence between X¹₁ and X¹₃ knowing that X¹₁ and X¹₂ are uncorrected. A locally best invariant test is proposed for this problem.     

Focused information criteria for copulas

In this paper, we extend the focused information criterion (FIC) to copula models. Copulas are often used for applications where the joint tail behavior of the variables is of particular interest, and selecting a copula that captures this well is then essential. Traditional model selection methods such as the Akaike information criterion (AIC) and the Bayesian information criterion (BIC) aim at finding the overall best‐fitting model, which is not necessarily the one best suited for the application at hand. The FIC, on the other hand, evaluates and ranks candidate models based on the precision of their point estimates of a context‐given focus parameter. This could be any quantity of particular interest, for example, the mean, a correlation, conditional probabilities, or measures of tail dependence. We derive FIC formulae for the maximum likelihood estimator, the two‐stage maximum likelihood estimator, and the so‐called pseudo‐maximum‐likelihood (PML) estimator combined with parametric margins. Furthermore, we confirm the validity of the AIC formula for the PML estimator combined with parametric margins. To study the numerical behavior of FIC, we have carried out a simulation study, and we have also analyzed a multivariate data set pertaining to abalones. The results from the study show that the FIC successfully ranks candidate models in terms of their performance, defined as how well they estimate the focus parameter. In terms of estimation precision, FIC clearly outperforms AIC, especially when the focus parameter relates to only a specific part of the model, such as the conditional upper‐tail probability.     

Efficient M-estimators with auxiliary information

This paper introduces a new class of M-estimators based on generalised empirical likelihood (GEL) estimation with some auxiliary information available in the sample. The resulting class of estimators is efficient in the sense that it achieves the same asymptotic lower bound as that of the efficient generalised method of moment (GMM) estimator with the same auxiliary information. The paper also shows that in case of smooth estimating equations the proposed estimators enjoy a small second order bias property compared to both efficient GMM and full GEL estimators. Analytical formulae to obtain bias corrected estimators are also provided. Simulations show that with correctly specified auxiliary information the proposed estimators and in particular those based on empirical likelihood outperform standard M and efficient GMM estimators both in terms of finite sample bias and efficiency. On the other hand with moderately misspecified auxiliary information estimators based on the nonparametric tilting method are typically characterised by the best finite sample properties.     

Optimal use of historical information

When historical data are available, incorporating them in an optimal way into the current data analysis can improve the quality of statistical inference. In Bayesian analysis, one can achieve this by using quality-adjusted priors of Zellner, or using power priors of Ibrahim and coauthors. These rules are constructed by raising the prior and/or the sample likelihood to some exponent values, which act as measures of compatibility of their quality or proximity of historical data to current data. This paper presents a general, optimum procedure that unifies these rules and is derived by minimizing a Kullback–Leibler divergence under a divergence constraint. We show that the exponent values are directly related to the divergence constraint set by the user and investigate the effect of this choice theoretically and also through sensitivity analysis. We show that this approach yields ‘100% efficient’ information processing rules in the sense of Zellner. Monte Carlo experiments are conducted to investigate the effect of historical and current sample sizes on the optimum rule. Finally, we illustrate these methods by applying them on real data sets.     

Fisher information in weighted distributions

Suppose that a density f_θ (x) belongs to an exponential family, but that inference about θ must be based on data that are obtained from a density that is proportional to W(x)f_θ(x). The authors study the Fisher information about θ in observations obtained from such weighted distributions and give conditions under which this information is greater than under the original density. These conditions involve the hazard- and reversed-hazard-rate functions.     

Marginal information for expectation parameters

The authors consider a special case of inference in the presence of nuisance parameters. They show that when the orthogonalized score function is a function of a statistic S, no Fisher information for the interest parameter is lost by using the marginal distribution of S rather than the full distribution of the observations. Therefore, no information for the interest parameter is recovered by conditioning on an ancillary statistic, and information will be lost by conditioning on an approximate ancillary statistic. This is the case for regular multivariate exponential families when the interest parameter is a subvector of the expectation parameter and the statistic is the maximum likelihood estimate of the interest parameter. Several examples are considered, including the 2 × 2 table.     

Contingency tables with fuzzy information

ABSTRACT

This paper extends the classical methods of analysis of a two-way contingency table to the fuzzy environment for two cases: (1) when the available sample of observations is reported as imprecise data, and (2) the case in which we prefer to categorize the variables based on linguistic terms rather than as crisp quantities. For this purpose, the α-cuts approach is used to extend the usual concepts of the test statistic and p-value to the fuzzy test statistic and fuzzy p-value. In addition, some measures of association are extended to the fuzzy version in order to evaluate the dependence in such contingency tables. Some practical examples are provided to explain the applicability of the proposed methods in real-world problems.     

Fisher information in generalized order statistics

A representation of the Fisher information in generalized order statistics in terms of the hazard rate of the underlying distribution function is derived under mild regularity conditions. This expression supplements results for complete, Type-II censored, and progressively Type-II censored data. As a byproduct, we find a hazard rate based representation for samples of k-records which apparently has not been known so far. Moreover, sufficient conditions for the validity of this representation in location and scale family settings are given. The result is illustrated by considering generalized order statistics based on logistic, Laplace, and extreme value distributions.     

Machine covering with combined partial information

Machine scheduling and covering problems may occur in many applications such as load balancing in network communication channel assignment, parallel processing in large-size computing, task arrangement in flexible manufacturing systems, etc. In this paper we study machine covering problems with combined partial information on m   parallel identical machines. We consider sequences where the processing time of all jobs are at most 1/k$1/k$ times of the optimal value (for an integer k). For the case where the optimal value is not known in advance, we show that LS algorithm is optimal. For the case where the optimal value is known in advance, we give lower bounds and present semi-online algorithms.     

Value of the beta prior information

Information gain or loss is studied here by considering changes in the reciprocal of the expected posterior variance. For two beta distributions, the ratio of the expected values of their posterior variances provides a convenient criterion which is consistent with several results of the bayesian set-up and also permits the determination of the least informative prior beta distribution. Relations with results obtained by other authors are also discussed.