Negative variance components for non-negative hierarchical data with correlation,over-, and/or underdispersion

The concept of negative variance components in linear mixed-effects models, while confusing at first sight, has received considerable attention in the literature, for well over half a century, following the early work of Chernoff 7 H. Chernoff, On the distribution of the likelihood ratio, Ann. Math. Statist. 25 (1954), pp. 573–578.Crossref] , Google Scholar]] and Nelder 21 J.A. Nelder, The interpretation of negative components of variance, Biometrika 41 (1954), pp. 544–548.Crossref], Web of Science ®] , Google Scholar]]. Broadly, negative variance components in linear mixed models are allowable if inferences are restricted to the implied marginal model. When a hierarchical view-point is adopted, in the sense that outcomes are specified conditionally upon random effects, the variance–covariance matrix of the random effects must be positive-definite (positive-semi-definite is also possible, but raises issues of degenerate distributions). Many contemporary software packages allow for this distinction. Less work has been done for generalized linear mixed models. Here, we study such models, with extension to allow for overdispersion, for non-negative outcomes (counts). Using a study of trichomes counts on tomato plants, it is illustrated how such negative variance components play a natural role in modeling both the correlation between repeated measures on the same experimental unit and over- or underdispersion.