Sufficient reduction methods for multivariate time-dependent health surveillance data

ABSTRACT

Early detection with a low false alarm rate (FAR) is the main aim of outbreak detection as used in public health surveillance or in regard to bioterrorism. Multivariate surveillance is preferable to univariate surveillance since correlation between series (CBS) is recognized and incorporated. Sufficient reduction has proved a promising method for handling CBS, but has not previously been used when correlation within series (CWS) is present. Here we develop sufficient reduction methods for reducing a p-dimensional multivariate series to a univariate series of statistics shown to be sufficient to monitor a sudden, but persistent, shift in the multivariate series mean. Correlation both within and between series is taken into account, as public health data typically exhibit both forms of association. Simultaneous and lagged changes and different shift sizes are investigated. A one-sided exponentially weighted moving average chart is used as a tool for detection of a change. The performance of the proposed method is compared with existing sufficient reduction methods, the parallel univariate method and both VarR and Z charts. A simulation study using bivariate normal autoregressive data shows that the new method gives shorter delays and a lower FAR than other methods, which have high FARs when CWS is clearly present.     

Effect of Dependency in Systems for Multivariate Surveillance

Systems for multivariate on-line surveillance (e.g., outbreak detection) are investigated. Optimal systems for statistical surveillance are based on likelihood ratios. Three systems are compared: based on each marginal density, based on the joint density, and based on the Hotelling's T2. The effect of dependency between the monitored processes is investigated, and the effect of correlation between the change times. When the first change occurs immediately, the three methods give similar delay of an alarm, in the situation with independency. For late changes, T2 has the longest delay, both for independent processes and for processes with a positive covariance.     

A Comparison of Likelihood-Based Spatiotemporal Monitoring Methods Under Non-Homogenous Population Size

This article discusses the spatiotemporal surveillance problem of detecting rate changes of Poisson data considering non-homogenous population sample size. By applying Monte Carlo simulations, we investigate the performance of several likelihood-based approaches under various scenarios depending on four factors: (1) population trend, (2) change magnitude, (3) change coverage, and (4) change time. Our article evaluates the performance of spatiotemporal surveillance methods based on the average run length at different change times. The simulation results show that no method is uniformly better than others in all scenarios. The difference between the generalized likelihood ratio (GLR) approach and the weighted likelihood ratio (WLR) approach depends mainly on population size, not change coverage, change magnitude, or change time. We find that changes associated with a small population in time periods and/or spatial regions favor the WLR approach, but those associated with a large population favor the GLR under any trends of population changes.     

A statistical approach to social network monitoring

Social network monitoring consists of monitoring changes in networks with the aim of detecting significant ones and attempting to identify assignable cause(s) contributing to the occurrence of a change. This paper proposes a method that helps to overcome some of the weaknesses of the existing methods. A Poisson regression model for the probability of the number of communications between network members as a function of vertex attributes is constructed. Multivariate exponentially weighted moving average (MEWMA) and multivariate cumulative sum (MCUSUM) control charts are used to monitor the network formation process. The results indicate more efficient performance for the MEWMA chart in identifying significant changes.     

Detecting clusters with increased mean using scan windows with variable radius

Applying spatiotemporal scan statistics is an effective method to detect the clustering of mean shifts in many application fields. Although several exponentially weighted moving average (EWMA) based scan statistics have been proposed, the existing methods generally require a fixed scan window size or apply the weighting technique across the temporal axis only. However, the size of shift coverage is often unavailable in practical problems. Using a mismatching scan radius may mislead the size of cluster coverage in space or delay the time to detection. This research proposed an stEWMA method by applying the weighting technique across both temporal and spatial axes with variable scan radius. The simulation analysis showed that the stEWMA method can have a significantly shorter time to detection than the likelihood ratio-based scan statistic using variable scan radius, especially when cluster coverage size is small. The application to detecting the increase of male thyroid cancer in the New Mexico state also showed the effectiveness of the proposed method.     

Some principles for surveillance adopted for multivariate processes with a common change point

The surveillance of multivariate processes has received growing attention during the last decade. Several generalizations of well-known methods such as Shewhart, CUSUM and EWMA charts have been proposed. Many of these multivariate procedures are based on a univariate summarized statistic of the multivariate observations, usually the likelihood ratio statistic. In this paper we consider the surveillance of multivariate observation processes for a shift between two fully specified alternatives. The effect of the dimension reduction using likelihood ratio statistics are discussed in the context of sufficiency properties. Also, an example of the loss of efficiency when not using the univariate sufficient statistic is given. Furthermore, a likelihood ratio method, the LR method, for constructing surveillance procedures is suggested for multivariate surveillance situations. It is shown to produce univariate surveillance procedures based on the sufficient likelihood ratios. As the LR procedure has several optimality properties in the univariate, it is also used here as a benchmark for comparisons between multivariate surveillance procedures     

A review and discussion of prospective statistical surveillance in public health

Summary. A review of methods suggested in the literature for sequential detection of changes in public health surveillance data is presented. Many researchers have noted the need for prospective methods. In recent years there has been an increased interest in both the statistical and the epidemiological literature concerning this type of problem. However, most of the vast literature in public health monitoring deals with retrospective methods, especially spatial methods. Evaluations with respect to the statistical properties of interest for prospective surveillance are rare. The special aspects of prospective statistical surveillance and different ways of evaluating such methods are described. Attention is given to methods that include only the time domain as well as methods for detection where observations have a spatial structure. In the case of surveillance of a change in a Poisson process the likelihood ratio method and the Shiryaev–Roberts method are derived.     

Multivariate outbreak detection

Online monitoring is needed to detect outbreaks of diseases such as influenza. Surveillance is also needed for other kinds of outbreaks, in the sense of an increasing expected value after a constant period. Information on spatial location or other variables might be available and may be utilized. We adapted a robust method for outbreak detection to a multivariate case. The relation between the times of the onsets of the outbreaks at different locations (or some other variable) was used to determine the sufficient statistic for surveillance. The derived maximum-likelihood estimator of the outbreak regression was semi-parametric in the sense that the baseline and the slope were non-parametric while the distribution belonged to the one-parameter exponential family. The estimator was used in a generalized-likelihood ratio surveillance method. The method was evaluated with respect to robustness and efficiency in a simulation study and applied to spatial data for detection of influenza outbreaks in Sweden.     

On the use and evaluation of prospective scan methods for health-related surveillance

Summary.  We review some prospective scan-based methods that are used in health-related applications to detect increased rates of mortality or morbidity and to detect bioterrorism or active clusters of disease. We relate these methods to the use of the moving average chart in industrial applications. Issues that are related to the performance evaluation of spatiotemporal scan-based methods are discussed. In particular we clarify the definition of a recurrence interval and demonstrate that this measure does not reflect some important aspects of the statistical performance of scan-based, and other, surveillance methods. Some research needs in this area are given.     

Pairwise likelihood estimation for multivariate mixed Poisson models generated by Gamma intensities

Estimating the parameters of multivariate mixed Poisson models is an important problem in image processing applications, especially for active imaging or astronomy. The classical maximum likelihood approach cannot be used for these models since the corresponding masses cannot be expressed in a simple closed form. This paper studies a maximum pairwise likelihood approach to estimate the parameters of multivariate mixed Poisson models when the mixing distribution is a multivariate Gamma distribution. The consistency and asymptotic normality of this estimator are derived. Simulations conducted on synthetic data illustrate these results and show that the proposed estimator outperforms classical estimators based on the method of moments. An application to change detection in low-flux images is also investigated.     

Nonparametric depth-based multivariate outlier identifiers, and masking robustness properties

In extending univariate outlier detection methods to higher dimension, various issues arise: limited visualization methods, inadequacy of marginal methods, lack of a natural order, limited parametric modeling, and, when using Mahalanobis distance, restriction to ellipsoidal contours. To address and overcome such limitations, we introduce nonparametric multivariate outlier identifiers based on multivariate depth functions, which can generate contours following the shape of the data set. Also, we study masking robustness, that is, robustness against misidentification of outliers as nonoutliers. In particular, we define a masking breakdown point (MBP), adapting to our setting certain ideas of Davies and Gather [1993. The identification of multiple outliers (with discussion). Journal of the American Statistical Association 88, 782–801] and Becker and Gather [1999. The masking breakdown point of multivariate outlier identification rules. Journal of the American Statistical Association 94, 947–955] based on the Mahalanobis distance outlyingness. We then compare four affine invariant outlier detection procedures, based on Mahalanobis distance, halfspace or Tukey depth, projection depth, and “Mahalanobis spatial” depth. For the goal of threshold type outlier detection, it is found that the Mahalanobis distance and projection procedures are distinctly superior in performance, each with very high MBP, while the halfspace approach is quite inferior. When a moderate MBP suffices, the Mahalanobis spatial procedure is competitive in view of its contours not constrained to be elliptical and its computational burden relatively mild. A small sampling experiment yields findings completely in accordance with the theoretical comparisons. While these four depth procedures are relatively comparable for the purpose of robust affine equivariant location estimation, the halfspace depth is not competitive with the others for the quite different goal of robust setting of an outlyingness threshold.     

Mixed multivariate EWMA-CUSUM control charts for an improved process monitoring

Multivariate exponential weighted moving average and cumulative sum charts are the most common memory type multivariate control charts. They make use of the present and past information to detect small shifts in the process parameter(s). In this article, we propose two new multivariate control charts using a mixed version of their design setups. The plotting statistics of the proposed charts are based on the cumulative sum of the multivariate exponentially weighted moving averages. The performances of these schemes are evaluated in terms of average run length. The proposals are compared with their existing counterparts, including HotellingT², MCUSUM, MEWMA, and MC1 charts. An application example is also presented for practical considerations using a real dataset.     

Linearly admissible estimators of mean vector with respect to balanced loss function in multivariate statistics

The authors discuss the bias of the estimate of the variance of the overall effect synthesized from individual studies by using the variance weighted method. This bias is proven to be negative. Furthermore, the conditions, the likelihood of underestimation and the bias from this conventional estimate are studied based on the assumption that the estimates of the effect are subject to normal distribution with common mean. The likelihood of underestimation is very high (e.g. it is greater than 85% when the sample sizes in two combined studies are less than 120). The alternative less biased estimates for the cases with and without the homogeneity of the variances are given in order to adjust for the sample size and the variation of the population variance. In addition, the sample size weight method is suggested if the consistence of the sample variances is violated Finally, a real example is presented to show the difference by using the above three estimate methods.     

An unconditional space–time scan statistic for ZIP‐distributed data

A scan statistic is proposed for the prospective monitoring of spatiotemporal count data with an excess of zeros. The method that is based on an outbreak model for the zero‐inflated Poisson distribution is shown to be superior to traditional scan statistics based on the Poisson distribution in the presence of structural zeros. The spatial accuracy and the detection timeliness of the proposed scan statistic are investigated by means of simulation, and an application on the weekly cases of Campylobacteriosis in Germany illustrates how the scan statistic could be used to detect emerging disease outbreaks. An implementation of the method is provided in the open‐source R package scanstatistics available on the Comprehensive R Archive Network.     

高维大数据基因网络中的社区发现——以NC方法为例

从大量基因中识别出致病基因是大数据下的一个十分重要的高维统计问题。基因间网络结构的存在使得对于致病基因的识别已从单个基因识别扩展到基因模块识别。从基因网络中挖掘出基因模块就是所谓的社区发现（或节点聚类）问题。绝大多数社区发现方法仅利用网络结构信息,而忽略节点本身的信息。Newman和Clauset于2016年提出了一个将二者有机结合的基于统计推断的社区发现方法（简称为NC方法）。本文以NC方法为案例,介绍统计方法在实际基因网络中的应用和取得的成果,并从统计学角度提出了改进措施。通过对NC方法的分析可以看出对于以基因网络为代表的非结构化数据,统计思想和原理在数据分析中仍然处于核心地位。而相应的统计方法则需要针对数据的特点及关心的问题进行相应的调整和优化。     

Structural change tests for GEL criteria

This article examines structural change tests based on generalized empirical likelihood methods in the time series context, allowing for dependent data. Standard structural change tests for the Generalized method of moments (GMM) are adapted to the generalized empirical likelihood (GEL) context. We show that when moment conditions are properly smoothed, these test statistics converge to the same asymptotic distribution as in the GMM, in cases with known and unknown breakpoints. New test statistics specific to GEL methods, and that are robust to weak identification, are also introduced. A simulation study examines the small sample properties of the tests and reveals that GEL-based robust tests performed well, both in terms of the presence and location of a structural change and in terms of the nature of identification.     

A comparison of statistical methods for animal oncology studies

In pre-clinical oncology studies, tumor-bearing animals are treated and observed over a period of time in order to measure and compare the efficacy of one or more cancer-intervention therapies along with a placebo/standard of care group. A data analysis is typically carried out by modeling and comparing tumor volumes, functions of tumor volumes, or survival. Data analysis on tumor volumes is complicated because animals under observation may be euthanized prior to the end of the study for one or more reasons, such as when an animal's tumor volume exceeds an upper threshold. In such a case, the tumor volume is missing not-at-random for the time remaining in the study. To work around the non-random missingness issue, several statistical methods have been proposed in the literature, including the rate of change in log tumor volume and partial area under the curve. In this work, an examination and comparison of the test size and statistical power of these and other popular methods for the analysis of tumor volume data is performed through realistic Monte Carlo computer simulations. The performance, advantages, and drawbacks of popular statistical methods for animal oncology studies are reported. The recommended methods are applied to a real data set.     

Evaluation of multivariate surveillance

Multivariate surveillance is of interest in many areas such as industrial production, bioterrorism detection, spatial surveillance, and financial transaction strategies. Some of the suggested approaches to multivariate surveillance have been multivariate counterparts to the univariate Shewhart, EWMA, and CUSUM methods. Our emphasis is on the special challenges of evaluating multivariate surveillance methods. Some new measures are suggested and the properties of several measures are demonstrated by applications to various situations. It is demonstrated that zero-state and steady-state ARL, which are widely used in univariate surveillance, should be used with care in multivariate surveillance.     

Robust multivariate change point analysis based on data depth

Modern methods for detecting changes in the scale or covariance of multivariate distributions rely primarily on testing for the constancy of the covariance matrix. These depend on higher-order moment conditions, and also do not work well when the dimension of the data is large or even moderate relative to the sample size. In this paper, we propose a nonparametric change point test for multivariate data using rankings obtained from data depth measures. As the data depth of an observation measures its centrality relative to the sample, changes in data depth may signify a change of scale of the underlying distribution, and the proposed test is particularly responsive to detecting such changes. We provide a full asymptotic theory for the proposed test statistic under the null hypothesis that the observations are stable, and natural conditions under which the test is consistent. The finite sample properties are investigated by means of a Monte Carlo simulation, and these along with the theoretical results confirm that the test is robust to heavy tails, skewness and high dimensionality. The proposed methods are demonstrated with an application to structural break detection in the rate of change of pollutants linked to acid rain measured in Turkey lake, a lake in central Ontario, Canada. Our test suggests a change in the rate of acid rain in the late 1980s/early 1990s, which coincides with clean air legislation in Canada and the US. The Canadian Journal of Statistics 48: 417–446; 2020 © 2020 Statistical Society of Canada     

Regularised Manova for High‐Dimensional Data

The traditional and readily available multivariate analysis of variance (MANOVA) tests such as Wilks' Lambda and the Pillai–Bartlett trace start to suffer from low power as the number of variables approaches the sample size. Moreover, when the number of variables exceeds the number of available observations, these statistics are not available for use. Ridge regularisation of the covariance matrix has been proposed to allow the use of MANOVA in high‐dimensional situations and to increase its power when the sample size approaches the number of variables. In this paper two forms of ridge regression are compared to each other and to a novel approach based on lasso regularisation, as well as to more traditional approaches based on principal components and the Moore‐Penrose generalised inverse. The performance of the different methods is explored via an extensive simulation study. All the regularised methods perform well; the best method varies across the different scenarios, with margins of victory being relatively modest. We examine a data set of soil compaction profiles at various positions relative to a ridgetop, and illustrate how our results can be used to inform the selection of a regularisation method.