Improving the detection of unusual observations in high‐dimensional settings

Multivariate control charts are used to monitor stochastic processes for changes and unusual observations. Hotelling's T² statistic is calculated for each new observation and an out‐of‐control signal is issued if it goes beyond the control limits. However, this classical approach becomes unreliable as the number of variables p approaches the number of observations n, and impossible when p exceeds n. In this paper, we devise an improvement to the monitoring procedure in high‐dimensional settings. We regularise the covariance matrix to estimate the baseline parameter and incorporate a leave‐one‐out re‐sampling approach to estimate the empirical distribution of future observations. An extensive simulation study demonstrates that the new method outperforms the classical Hotelling T² approach in power, and maintains appropriate false positive rates. We demonstrate the utility of the method using a set of quality control samples collected to monitor a gas chromatography–mass spectrometry apparatus over a period of 67 days.     

When and how to satisfice: an experimental investigation

This paper is about satisficing behaviour. Rather tautologically, this is when decision-makers are satisfied with achieving some objective, rather than in obtaining the best outcome. The term was coined by Simon (Q J Econ 69:99–118, 1955), and has stimulated many discussions and theories. Prominent amongst these theories are models of incomplete preferences, models of behaviour under ambiguity, theories of rational inattention, and search theories. Most of these, however, seem to lack an answer to at least one of two key questions: when should the decision-maker (DM) satisfice; and how should the DM satisfice. In a sense, search models answer the latter question (in that the theory tells the DM when to stop searching), but not the former; moreover, usually the question as to whether any search at all is justified is left to a footnote. A recent paper by Manski (Theory Decis. doi: 10.1007/s11238-017-9592-1, 2017) fills the gaps in the literature and answers the questions: when and how to satisfice? He achieves this by setting the decision problem in an ambiguous situation (so that probabilities do not exist, and many preference functionals can therefore not be applied) and by using the Minimax Regret criterion as the preference functional. The results are simple and intuitive. This paper reports on an experimental test of his theory. The results show that some of his propositions (those relating to the ‘how’) appear to be empirically valid while others (those relating to the ‘when’) are less so.     

R×C ecological inference: bounds, correlations, flexibility and transparency of assumptions

Summary.  Despite its potential pitfalls, ecological inference is an unavoidable part of some quantitative settings, including US voting rights litigation. In such applications, the analyst will typically encounter two-way tables with more than two rows and columns. Although several ecological inference methods are currently available for 2×2 tables, there are fewer options for analysing general R × C tables, and virtually none that model counts as opposed to fractions. We propose a count R × C method that respects the bounds deterministically, that allows for complex relationships between internal cell quantities, that is easily extensible and that results from transparent assumptions. We study the method via simulation, and then apply it to an example that is drawn from the state of Texas relevant to recent redistricting litigation there.     

A new estimation method for Weibull-type tails based on the mean excess function

Studying the right tail of a distribution, one can classify the distributions into three classes based on the extreme value index γ$\gamma$. The class γ>0$\gamma >0$ corresponds to Pareto-type or heavy tailed distributions, while γ<0$\gamma <0$ indicates that the underlying distribution has a finite endpoint. The Weibull-type distributions form an important subgroup within the Gumbel class with γ=0$\gamma =0$. The tail behaviour can then be specified using the Weibull tail index. Classical estimators of this index show severe bias. In this paper we present a new estimation approach based on the mean excess function, which exhibits improved bias and mean squared error. The asserted properties are supported by simulation experiments and asymptotic results. Illustrations with real life data sets are provided.     

THE USE OF AGGREGATE DATA TO ESTIMATE GOMPERTZ‐TYPE OLD‐AGE MORTALITY IN HETEROGENEOUS POPULATIONS

We consider two related aspects of the study of old‐age mortality. One is the estimation of a parameterized hazard function from grouped data, and the other is its possible deceleration at extreme old age owing to heterogeneity described by a mixture of distinct sub‐populations. The first is treated by half of a logistic transform, which is known to be free of discretization bias at older ages, and also preserves the increasing slope of the log hazard in the Gompertz case. It is assumed that data are available in the form published by official statistical agencies, that is, as aggregated frequencies in discrete time. Local polynomial modelling and weighted least squares are applied to cause‐of‐death mortality counts. The second, related, problem is to discover what conditions are necessary for population mortality to exhibit deceleration for a mixture of Gompertz sub‐populations. The general problem remains open but, in the case of three groups, we demonstrate that heterogeneity may be such that it is possible for a population to show decelerating mortality and then return to a Gompertz‐like increase at a later age. This implies that there are situations, depending on the extent of heterogeneity, in which there is at least one age interval in which the hazard function decreases before increasing again.     

Shrinkage inverse regression estimation for model-free variable selection

Summary.  The family of inverse regression estimators that was recently proposed by Cook and Ni has proven effective in dimension reduction by transforming the high dimensional predictor vector to its low dimensional projections. We propose a general shrinkage estimation strategy for the entire inverse regression estimation family that is capable of simultaneous dimension reduction and variable selection. We demonstrate that the new estimators achieve consistency in variable selection without requiring any traditional model, meanwhile retaining the root n estimation consistency of the dimension reduction basis. We also show the effectiveness of the new estimators through both simulation and real data analysis.     

VARIATIONAL BAYESIAN ANALYSIS FOR HIDDEN MARKOV MODELS

The variational approach to Bayesian inference enables simultaneous estimation of model parameters and model complexity. An interesting feature of this approach is that it also leads to an automatic choice of model complexity. Empirical results from the analysis of hidden Markov models with Gaussian observation densities illustrate this. If the variational algorithm is initialized with a large number of hidden states, redundant states are eliminated as the method converges to a solution, thereby leading to a selection of the number of hidden states. In addition, through the use of a variational approximation, the deviance information criterion for Bayesian model selection can be extended to the hidden Markov model framework. Calculation of the deviance information criterion provides a further tool for model selection, which can be used in conjunction with the variational approach.     

The Effect of Population Growth on Economic Growth: A Meta‐Regression Analysis of the Macroeconomic Literature

Many studies have sought to gauge the impact of population growth on economic growth. A well‐known stylized fact of this literature is that the estimated effects of population growth measures on economic growth are not robust, varying between being positive, negative, and insignificantly different from zero. The present study analyzes 471 statistical regressions from 29 prominent economic growth studies using meta‐regression analysis to identify the effect of alternative methodologies on key population growth results. This study finds that a broad set of methodological factors explains more than half of the variation in the population growth effects observed from this literature, including the types of variables used to measure population growth, the countries selected, the time frame of the analysis, and the nature of the control variables specified. The study also yields results that have implications for policymakers, especially insofar as several policy factors seem to influence the population change–economic growth nexus. Particularly strong is the evidence in support of the increasingly adverse effects of population growth in the post‐1980 period, suggesting that demographic issues should warrant greater attention than they currently receive from the policymaking community.     

Racial Disparities in Early Criminal Justice Involvement

Criminologists have long reported the existence of racial disparity in the criminal justice system, but the important question is why. While some argue that observed differences are a consequence of more criminal behavior among minorities, the weight of the evidence indicates that this is but a partial explanation. In this paper, we study data from a sample of juveniles to examine how racial differences in early police contact, and important social environments—family, school, and neighborhoods—affect later contact and arrests, controlling for self-reported delinquency. We find that early (in middle school) contact with police is an important predictor of later (high school) arrests. Also we found that, in addition to being male and living in a low-income family, children who have parents who have a history of arrest, who have experienced school disciplinary actions, who have delinquent peers, and who are in networks with deviant adults are more likely to have problems with law enforcement. These factors help to explain racial differences in police contacts and arrests.