Significance test of clustering under high dimensional setting with applications to cancer data

For high dimensional data, the SigClust is developed for testing the significance of clustering. The cluster index (CI) for SigClust is conducted by the ratio of the within-cluster and total sum of squares. But its empirical size is too conservative to be over controlled. By removing the cumbrous terms in the CI, an improved index (BCI) is proposed in this paper. The coefficient of variation of the BCI can be significantly reduced, implying that the new index BCI is stable. Moreover, the new significance test (NewSig) maintains the size, meanwhile, provides a greater power. Simulation experiments and two real cancer data examples are analysed for illustrating the performance of the new methodology.